An Observation and Control Method and System for Langmuir Circulation

Abstract

The present invention discloses an observation and control method and system for Langmuir circulation. The method includes: configuring an unmanned boat to a target observation area according to the destination location information of a target observation task; releasing a dye tracer into the water body through the unmanned boat; starting from the location where the unmanned boat is located, configuring an unmanned aerial vehicle to follow the diffusion path of the dye tracer for image acquisition; performing image recognition on the acquired images to determine whether it is Langmuir circulation; when Langmuir circulation is recognized, configuring the data acquisition route of the unmanned boat according to the acquired images, controlling the unmanned boat formation to detect the Langmuir circulation area according to the data acquisition route, and performing image acquisition and recognition on the Langmuir circulation area through an unmanned aerial vehicle cluster to complete the edge detection of the Langmuir circulation area. The observation process of the present invention is flexible, mobile, low-cost and high-efficiency, and can be widely applied to the field of environmental monitoring technology.

Description

Technical Field

[0001] The present invention relates to the technical field of environmental monitoring, and in particular to an observation control method and system for Langmuir circulation. Background Art

[0002] Ocean circulation studies the sea currents caused by winds, density currents generated by uneven density distribution, the generation and distribution of eddies in the ocean circulation, the westward intensification of the ocean circulation, the bending and variation of sea currents, the flow system structure in the near-equatorial region, the Antarctic Circumpolar Current, the ocean thermohaline circulation, the deep-sea circulation and its relationship with the main thermocline, the relationship between the divergence and convergence movements of seawater and upwelling and downwelling currents and Langmuir circulation, mesoscale eddies and their energy conversion, special flow phenomena such as ice drift, the response of the ocean to wind stress, etc., as well as the circulation in the coastal sea area and so on.

[0003] Compared with the atmospheric circulation, the observation of ocean circulation is more difficult and the data is more scarce.

[0004] Langmuir circulation is a longitudinal spiral vortex motion occurring in the upper layer of the ocean, and its axial direction is basically consistent with the wind direction. Upwelling or downwelling currents will be generated at adjacent two vortices. The downwelling current will cause the aggregation of surface floating objects to form strip-shaped traces that are basically parallel to the wind direction. Langmuir circulation is a common and important ocean phenomenon, and Langmuir circulation has a very close relationship with many important issues in ocean research. On the one hand, Langmuir circulation will bring heat, substances and momentum in the upper mixed layer to the deep layer, and this transmission effect not only enhances ocean mixing but also affects the material and energy exchange between the ocean and the atmosphere. On the other hand, the mixing it causes in the upper layer of the ocean also redistributes nutrients and plankton, etc. and generates special ecological regions within the water layer. These have significant implications for the research of the ocean environment and ecology, etc. In addition, it also has important practical significance in aspects such as ocean risk prevention and control, such as the harm prediction and effective recovery of oil spills, the prevention and control of toxic algae, etc. Therefore, it is very necessary to monitor Langmuir circulation.

[0005] The current observation method of Langmuir circulation still stays in the traditional method of using manned ships to carry measurement equipment for aerial surveys. This traditional method is not flexible enough, has high operation and maintenance costs and low efficiency. Summary of the Invention

[0006] In view of this, an embodiment of the present invention provides an observation control method and system for Langmuir circulation that is flexible, low-cost and high-efficiency.

[0007] The first aspect of the present invention provides an observation control method for Langmuir circulation, including:

[0008] Configuring an unmanned boat to a target observation area according to the destination location information of the target observation task;

[0009] Releasing a dye tracer into the water body through the unmanned boat;

[0010] Taking the location of the unmanned boat as the starting point, configuring an unmanned aerial vehicle (UAV) to follow the diffusion path of the dye tracer for image acquisition;

[0011] Performing image recognition on the acquired images to determine whether it is a Langmuir circulation;

[0012] When a Langmuir circulation is recognized, configuring the data acquisition route of the unmanned boat according to the acquired images, controlling the unmanned boat formation to detect the Langmuir circulation area according to the data acquisition route, and performing image acquisition and recognition on the Langmuir circulation area through a UAV cluster to complete the edge detection of the Langmuir circulation area.

[0013] Optionally, the releasing a dye tracer into the water body through the unmanned boat includes:

[0014] When the unmanned boat reaches the target observation area, dispatching a UAV from the position of the unmanned boat, and the UAV flies straight up into the air to take a water surface image;

[0015] When the number of bubble marks captured by the UAV is greater than or equal to the first threshold, hovering the UAV at the current altitude;

[0016] Controlling the unmanned boat to reach the middle position between two strip-shaped bubble marks according to the guidance of the UAV, and the unmanned boat releases the dye tracer from the water surface and underwater at the current position.

[0017] Optionally, the taking the location of the unmanned boat as the starting point, configuring a UAV to follow the diffusion path of the dye tracer for image acquisition includes:

[0018] Controlling the UAV to perform aerial operations and taking images through the UAV aerial platform;

[0019] Transmitting the images captured by the UAV through the boat-aircraft transmission system and storing them on the unmanned boat;

[0020] After processing the stored images through the unmanned boat data processing platform, planning the flight path of the UAV and sending it to the UAV aerial platform to control the UAV to further acquire images according to the flight path.

[0021] Optionally, the performing image recognition on the acquired images to determine whether it is a Langmuir circulation includes:

[0022] Obtaining the images to be trained, where the images to be trained include Langmuir circulation bubble marks in the Internet, strip-shaped traces formed by the diffusion of oil spills at sea following the circulation, images acquired after each execution of the circulation detection task, and image frames obtained by splitting the videos captured by the UAV;

[0023] The to-be-trained image is divided into a training set, a validation set, and a test set by means of stratified random sampling;

[0024] The training set is subjected to image annotation processing and data augmentation processing to obtain training set images;

[0025] The YOLOv5 algorithm based on transfer learning is used to perform deep learning on the training set images to construct a Langmuir circulation bubble mark recognition model;

[0026] According to the Langmuir circulation bubble mark recognition model, the collected images are subjected to image recognition to determine whether it is a Langmuir circulation.

[0027] Optionally, the data augmentation processing of the training set includes at least one of the following:

[0028] The training set is subjected to data flipping to obtain the first data;

[0029] Alternatively, the training set is subjected to data rotation to obtain the second data;

[0030] Alternatively, the training set is subjected to data scaling to obtain the third data;

[0031] Alternatively, the training set is subjected to data cropping to obtain the fourth data;

[0032] Alternatively, the training set is subjected to data translation to obtain the fifth data;

[0033] Alternatively, the training set is subjected to noise addition processing to obtain the sixth data.

[0034] Optionally, the using the YOLOv5 algorithm based on transfer learning to perform deep learning on the training set images to construct a Langmuir circulation bubble mark recognition model includes:

[0035] Knowledge transfer is performed on the object detection model pre-trained on the YOLOv5 for the coco dataset, and the Langmuir circulation bubble mark image dataset is trained to obtain the YOLOv5 Langmuir circulation bubble mark recognition model;

[0036] Then, knowledge transfer is performed on the YOLOv5 Langmuir circulation bubble mark recognition model to train the Langmuir circulation bubble mark recognition model after staining.

[0037] Optionally, the according to the Langmuir circulation bubble mark recognition model to perform image recognition on the collected images to determine whether it is a Langmuir circulation includes:

[0038] The collected images are subjected to color restoration processing to generate a color restoration matrix set;

[0039] Perform color deviation correction and restoration on the collected image according to the set of color restoration matrices;

[0040] Use the Langmuir circulation bubble mark recognition model to perform real-time recognition on the restored image, classify and determine the dyed red water body and the undyed blue-green water body, and further realize the recognition of the Langmuir circulation.

[0041] Optionally, when the Langmuir circulation is recognized, configure the data acquisition route of the unmanned boat according to the collected image, control the unmanned boat formation to detect the Langmuir circulation area according to the data acquisition route, and perform image acquisition and recognition on the Langmuir circulation area through the UAV cluster, and complete the edge detection of the Langmuir circulation area, including:

[0042] Perform grayscale processing on the collected image by the maximum value method to obtain a grayscale image;

[0043] Perform mean filtering on the grayscale image by the mean filtering algorithm to complete the sliding process of the grayscale image;

[0044] Slide the filtering window to the target pixel point, calculate the average value of the neighborhood pixel points of the pixel point, and replace the value of the pixel point with the average value of the neighborhood pixel points until the denoising process of all pixel points is completed;

[0045] Perform global binarization on the denoised grayscale image and then perform canny edge detection to detect the edge of the bubble mark and complete the edge detection of the Langmuir circulation area.

[0046] Optionally, there are three observation modes for the unmanned boat, including the mode of observing along the right-angle path of the edge, the mode of observing along the path perpendicular to the bubble mark, and the mode of observing along the navigation path of the dyeing strip;

[0047] Among them, the mode of observing along the right-angle path of the edge includes the following steps:

[0048] Search for and cluster the convex hull points of the graph to determine the target convex hull points;

[0049] Set a partition reference distance, and determine the category of the target convex hull points according to the number of partitions greater than the partition reference distance;

[0050] Determine the position of the offset line segment according to the position of the unmanned boat;

[0051] Find out the first line segment that does not cross the bubble trace area after offset, and define the intersection point of the line segment and the circle as the target circle point;

[0052] Connect the initial point and the target convex hull point to the target circular point respectively, obtaining a first path from the initial point to the target point, and the included angle between the two line segments forming the first path is a right angle;

[0053] Find the target circular point and determine a second path from the target point to the initial point;

[0054] Connect the intersection points of the first path and the second path to the initial point and the target convex hull point respectively, then the final path observed by the unmanned boat is obtained;

[0055] The mode of observing along the vertical bubble trace path includes the following steps:

[0056] The UAV provides real-time images of large-scale bubble traces in the air, and edge detection is used to detect the edges of the bubble traces;

[0057] Approximate the zigzag bubble trace as a straight line strip with width;

[0058] Determine the path of the unmanned boat according to the relative positions of the two approximately straight line strip bubble traces; among them, when the two bubble traces are parallel, the path of the unmanned boat adopts a path perpendicular to the starting bubble trace; when the two bubble traces are unbalanced and there is an included angle in the distance, take the position of the intersection point of the two bubble traces as the center of the circle, and use the concentric arc between the two bubble traces as the observation path of the unmanned boat;

[0059] Perform real-time planning on the observation path of the unmanned boat according to the real-time observation of the UAV. When the position of the bubble trace changes, optimize the path of the unmanned boat;

[0060] The mode of observing along the dyed strip navigation path includes the following steps:

[0061] Use the video acquisition module on the unmanned boat to track the dyed trace, actively adjust the camera parameters and posture, and perform image acquisition;

[0062] Adopt fuzzy control rules to control the rotation of the servo steering gear at the bottom of the camera bracket;

[0063] The camera continuously changes as the position of the dyed strip changes, making the trace located in the center of the image.

[0064] Optionally, the method further includes the step of stitching and fusing the images of all UAVs to obtain the full-region image of the Langmuir circulation generation area, and then estimating the area size of the Langmuir circulation generation area through geometric relationships. This step specifically includes:

[0065] Establish a UAV detection model to determine the detection range and corresponding geometric position relationships;

[0066] Establish an image sensor model;

[0067] After obtaining the full-region image of the Langmuir circulation generation area according to the UAV detection model and the image sensor model, image registration is first performed, and then image fusion is performed to obtain a regional panoramic view;

[0068] After obtaining the regional panoramic view of the Langmuir circulation generation area, the range of the Langmuir circulation generation area is estimated through geometric relationships.

[0069] Another aspect of the embodiments of the present invention also provides an observation and control system for Langmuir circulation, including:

[0070] A first module for configuring an unmanned boat to a target observation area according to the destination location information of the target observation task;

[0071] A second module for releasing a dye tracer into the water body through the unmanned boat;

[0072] A third module for starting from the location of the unmanned boat, configuring a UAV to follow the diffusion path of the dye tracer for image acquisition;

[0073] A fourth module for performing image recognition on the acquired images to determine whether it is a Langmuir circulation;

[0074] A fifth module for, when a Langmuir circulation is recognized, configuring the data acquisition route of the unmanned boat according to the acquired images, controlling the unmanned boat formation to detect the Langmuir circulation area according to the data acquisition route, and performing image acquisition and recognition on the Langmuir circulation area through a UAV cluster to complete the edge detection of the Langmuir circulation area.

[0075] Another aspect of the embodiments of the present invention also provides an electronic device, including a processor and a memory;

[0076] The memory is used to store programs;

[0077] The processor executes the program to implement the method as described above.

[0078] Another aspect of the embodiments of the present invention also provides a computer-readable storage medium, where the storage medium stores a program, and the program is executed by a processor to implement the method as described above.

[0079] The embodiments of the present invention also disclose a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method described above.

[0080] In an embodiment of the present invention, according to the destination location information of the target observation task, an unmanned boat is configured to the target observation area; a dye tracer is released into the water body through the unmanned boat; starting from the location where the unmanned boat is located, a drone is configured to follow the diffusion path of the dye tracer for image acquisition; the acquired images are subjected to image recognition to determine whether it is a Langmuir circulation; when a Langmuir circulation is recognized, according to the acquired images, a data acquisition route of the unmanned boat is configured, and the unmanned boat formation is controlled to detect the Langmuir circulation area according to the data acquisition route, and the drone cluster is used to perform image acquisition and recognition on the Langmuir circulation area to complete the edge detection of the Langmuir circulation area. The observation process of the present invention is flexible, low-cost and highly efficient. Description of the Drawings

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0082] Figure 1 It is a flowchart of the Langmuir circulation detection method provided by the embodiment of the present invention;

[0083] Figure 2 It is a generalized layout diagram of the observation equipment provided by the embodiment of the present invention;

[0084] Figure 3 It is a schematic diagram of the drone Langmuir circulation bubble trace tracking system provided by the embodiment of the present invention;

[0085] Figure 4 It is a flowchart of the image recognition provided by the embodiment of the present invention;

[0086] Figure 5 It is a flowchart of the Langmuir circulation recognition provided by the embodiment of the present invention;

[0087] Figure 6 It is a schematic diagram of the information structure of the annotation file provided by the embodiment of the present invention;

[0088] Figure 7 It is a flowchart of the multiple transfer learning process of the dye Langmuir circulation bubble trace recognition model provided by the embodiment of the present invention;

[0089] Figure 8 It is a flowchart of the bubble trace edge recognition provided by the embodiment of the present invention;

[0090] Figure 9 It is a flowchart of the generation of the right-angle path along the edge provided by the embodiment of the present invention;

[0091] Figure 10 Flow chart for generating the vertical bubble mark path provided by an embodiment of the present invention;

[0092] Figure 11 Flow chart for generating the navigation path along the dyeing strip provided by an embodiment of the present invention;

[0093] Figure 12 Organizational structure diagram of the UAV network provided by an embodiment of the present invention;

[0094] Figure 13 Schematic diagram of the UAV image sensor model provided by an embodiment of the present invention;

[0095] Figure 14 Flow chart for grouping and splicing 6 segmented images provided by an embodiment of the present invention. Detailed implementation manners

[0096] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0097] Aiming at the problems existing in the prior art, the present application proposes an observation method for Langmuir circulation in order to solve the defects existing in the prior art. Among them, an unmanned boat is an unmanned remotely controlled surface ship, which has the characteristics of small size, high speed and low cost. Compared with manned ships, unmanned boats can achieve autonomous driving through an intelligent control system, have strong adaptability to harsh marine environments, can penetrate into dangerous waters to perform tasks without taking the risk of personnel casualties, thereby broadening the operation scope of waterborne ships and covering the entire sea area. In addition, unmanned boats can work for a long time without replenishment, can conduct 24-hour dynamic monitoring of sea area use, avoid wasting funds due to frequent deployment of ships and personnel, do not require a personnel support system, and can carry more mission payloads. The system and equipment carried by the unmanned boat are used to detect the sub-surface structure of the Langmuir circulation, and the UAV is used to identify the strip-shaped traces formed by the Langmuir circulation on the sea surface to judge the scale of the Langmuir circulation, etc. At the same time, a ground wave radar observation system can be additionally carried on the unmanned boat to conduct real-time large-scale and refined observation of the wind, wave and current fields on the sea surface. Through the joint observation of the UAV and the unmanned boat, different detection equipment is carried to observe the 3D structure of the Langmuir circulation from multiple levels, from the air to the sea surface and then to the sub-surface of the sea surface, thereby greatly enriching the research data of the Langmuir circulation and providing technical support for sea surface oil spill treatment, maritime personnel search and rescue, marine algae pollution, etc.

[0098] Specifically, the first aspect of the present invention provides an observation control method for Langmuir circulation, including:

[0099] Configure the unmanned boat to the target observation area according to the destination location information of the target observation task;

[0100] Release a dye tracer into the water body through the unmanned boat;

[0101] Starting from the location of the unmanned boat, configure an unmanned aerial vehicle to follow the diffusion path of the dye tracer for image acquisition;

[0102] Perform image recognition on the acquired images to determine whether it is a Langmuir circulation;

[0103] When a Langmuir circulation is recognized, configure the data acquisition route of the unmanned boat according to the acquired images, control the unmanned boat formation to detect the Langmuir circulation area according to the data acquisition route, and perform image acquisition and recognition on the Langmuir circulation area through an unmanned aerial vehicle cluster to complete the edge detection of the Langmuir circulation area.

[0104] Optionally, the releasing a dye tracer into the water body through the unmanned boat includes:

[0105] When the unmanned boat reaches the target observation area, dispatch an unmanned aerial vehicle from the location of the unmanned boat. The unmanned aerial vehicle flies straight up into the air and takes a water surface image;

[0106] When the number of bubble marks captured by the unmanned aerial vehicle is greater than or equal to the first threshold, hover the unmanned aerial vehicle at the current altitude;

[0107] Under the guidance of the unmanned aerial vehicle, control the unmanned boat to reach the middle position between two strip-shaped bubble marks, and the unmanned boat releases the dye tracer from the water surface and underwater at the current position.

[0108] Optionally, the configuring an unmanned aerial vehicle to follow the diffusion path of the dye tracer for image acquisition starting from the location of the unmanned boat includes:

[0109] Control the unmanned aerial vehicle to perform aerial operations through an unmanned aerial vehicle platform and take images;

[0110] Transmit the images captured by the unmanned aerial vehicle through a boat-aircraft transmission system and store them on the unmanned boat;

[0111] After processing the stored images through an unmanned boat data processing platform, plan the flight path of the unmanned aerial vehicle and send it to the unmanned aerial vehicle platform to control the unmanned aerial vehicle to further acquire images according to the flight path.

[0112] Optionally, the performing image recognition on the acquired images to determine whether it is a Langmuir circulation includes:

[0113] Obtain the images to be trained, where the images to be trained include Langmuir circulation bubble marks in the Internet, strip-shaped marks formed by the diffusion of oil spills at sea along with the circulation, images collected after each execution of the circulation detection task, and image frames obtained by splitting the videos taken by drones;

[0114] Divide the images to be trained into a training set, a validation set, and a test set by using the method of stratified random sampling;

[0115] Perform image annotation processing and data augmentation processing on the training set to obtain training set images;

[0116] Use the YOLOv5 algorithm based on transfer learning to perform deep learning on the training set images to construct a Langmuir circulation bubble mark recognition model;

[0117] Perform image recognition on the collected images according to the Langmuir circulation bubble mark recognition model to determine whether it is a Langmuir circulation.

[0118] Optionally, the data augmentation processing on the training set includes at least one of the following:

[0119] Perform data flipping on the training set to obtain the first data;

[0120] Or, perform data rotation on the training set to obtain the second data;

[0121] Or, perform data scaling on the training set to obtain the third data;

[0122] Or, perform data cropping on the training set to obtain the fourth data;

[0123] Or, perform data translation on the training set to obtain the fifth data;

[0124] Or, perform noise addition processing on the training set to obtain the sixth data.

[0125] Optionally, the use of the YOLOv5 algorithm based on transfer learning to perform deep learning on the training set images to construct a Langmuir circulation bubble mark recognition model includes:

[0126] Perform knowledge transfer on the object detection model pre-trained on the YOLOv5 for the coco dataset, and train the Langmuir circulation bubble mark image dataset to obtain the YOLOv5 Langmuir circulation bubble mark recognition model;

[0127] Then perform knowledge transfer on the YOLOv5 Langmuir circulation bubble mark recognition model to train the Langmuir circulation bubble mark recognition model after staining.

[0128] Optionally, performing image recognition on the acquired image according to the Langmuir circulation bubble mark recognition model to determine whether it is a Langmuir circulation, including:

[0129] Performing color restoration processing on the acquired image to generate a set of color restoration matrices;

[0130] Performing color deviation correction and restoration on the acquired image according to the set of color restoration matrices;

[0131] Using the Langmuir circulation bubble mark recognition model to perform real-time recognition on the restored image, classifying and determining the dyed red water body and the undyed blue-green water body, and further realizing the recognition of the Langmuir circulation.

[0132] Optionally, when the Langmuir circulation is recognized, configuring the data acquisition route of the unmanned boat according to the acquired image, controlling the unmanned boat formation to detect the Langmuir circulation area according to the data acquisition route, and performing image acquisition and recognition on the Langmuir circulation area through the UAV cluster to complete the edge detection of the Langmuir circulation area, including:

[0133] Performing grayscale processing on the acquired image by the maximum value method to obtain a grayscale image;

[0134] Performing mean filtering on the grayscale image by the mean filtering algorithm to complete the sliding processing of the grayscale image;

[0135] Sliding the filtering window to the target pixel point, calculating the average value of the neighborhood pixel points of the pixel point, and replacing the value of the pixel point with the average value of the neighborhood pixel points until the denoising processing of all pixel points is completed;

[0136] Performing global binarization processing on the denoised grayscale image and then performing canny edge detection to detect the edge of the bubble mark and complete the edge detection of the Langmuir circulation area.

[0137] Optionally, there are three observation modes for the unmanned boat, including the mode of observing along the right-angle path of the edge, the mode of observing along the path perpendicular to the bubble mark, and the mode of observing along the navigation path of the dyeing strip;

[0138] Among them, the mode of observing along the right-angle path of the edge includes the following steps:

[0139] Searching for and clustering the convex hull points of the graph to determine the target convex hull points;

[0140] Setting a partition reference distance, and determining the category of the target convex hull points according to the number of partitions greater than the partition reference distance;

[0141] Determining the position of the offset line segment according to the position of the unmanned boat;

[0142] Find the first line segment that does not cross the bubble trace area after offset, and define the intersection point of this line segment and the circle as the target dot;

[0143] Connect the initial point and the target convex hull point to the target dot respectively, to obtain a first path from the initial point to the target point, and the included angle between the two line segments forming the first path is a right angle;

[0144] Find the target dot and determine a second path from the target point to the initial point;

[0145] Connect the intersection points of the first path and the second path to the initial point and the target convex hull point respectively, then the final path observed by the unmanned boat is obtained;

[0146] The mode of observing along the vertical bubble trace path includes the following steps:

[0147] The UAV provides real-time images of a large range of bubble traces in the air, and uses edge detection to detect the edges of the bubble traces;

[0148] Approximate the zigzag bubble trace as a straight line strip with a width;

[0149] Determine the path of the unmanned boat according to the relative positions of the two approximately straight line strip bubble traces; among them, when the two bubble traces are parallel, the path of the unmanned boat adopts a path perpendicular to the starting bubble trace; when the two bubble traces are not balanced and there is an included angle in the distance, take the position of the intersection point of the two bubble traces as the center of the circle, and use concentric arcs between the two bubble traces as the observation path of the unmanned boat;

[0150] Perform real-time planning on the observation path of the unmanned boat according to the real-time observation of the UAV, and optimize the path of the unmanned boat when the position of the bubble trace changes;

[0151] The mode of observing along the navigation path of the dyeing strip includes the following steps:

[0152] Use the video acquisition module on the unmanned boat to track the dyeing trace, actively adjust the camera parameters and attitude, and perform image acquisition;

[0153] Adopt fuzzy control rules to control the rotation of the servo steering gear at the bottom of the camera bracket;

[0154] The camera continuously changes as the position of the dyeing strip changes, so that the trace is located in the center of the image.

[0155] Optionally, the method further includes the step of stitching and fusing the images of all UAVs to obtain a full-area image of the Langmuir circulation generation area, and then estimating the area size of the Langmuir circulation generation area through geometric relations. This step specifically includes:

[0156] Establish a UAV detection model to determine the detection range and the corresponding geometric position relationship;

[0157] Establish an image sensor model;

[0158] According to the UAV detection model and the image sensor model, after obtaining the full-region image of the Langmuir circulation generation area, first perform image registration, and then perform image fusion to obtain a regional panoramic view;

[0159] After obtaining the regional panoramic view of the Langmuir circulation generation area, estimate the range of the Langmuir circulation generation area through geometric relationships.

[0160] Another aspect of the embodiments of the present invention also provides an observation and control system for Langmuir circulation, including:

[0161] The first module is used to configure the unmanned boat to the target observation area according to the destination position information of the target observation task;

[0162] The second module is used to release a dye tracer into the water body through the unmanned boat;

[0163] The third module is used to start from the location of the unmanned boat and configure a UAV to follow the diffusion path of the dye tracer for image acquisition;

[0164] The fourth module is used to perform image recognition on the acquired images to determine whether it is a Langmuir circulation;

[0165] The fifth module is used to, when a Langmuir circulation is recognized, configure the data acquisition route of the unmanned boat according to the acquired images, control the unmanned boat formation to detect the Langmuir circulation area according to the data acquisition route, and perform image acquisition and recognition on the Langmuir circulation area through a UAV cluster to complete the edge detection of the Langmuir circulation area.

[0166] Another aspect of the embodiments of the present invention also provides an electronic device, including a processor and a memory;

[0167] The memory is used to store programs;

[0168] The processor executes the program to implement the method as described above.

[0169] Another aspect of the embodiments of the present invention also provides a computer-readable storage medium, the storage medium stores a program, and the program is executed by a processor to implement the method as described above.

[0170] Embodiments of the present invention also disclose a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device may read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the foregoing method.

[0171] The following will describe in detail the specific implementation process of the present invention with reference to the accompanying drawings of the specification:

[0172] In this embodiment, as Figure 1 shown, a system for observing Langmuir circulation and an overall framework flowchart of the solution are provided. The general layout diagram of the observation equipment is as Figure 2 shown, and the specific steps are as follows:

[0173] In the first step, the sail unmanned boat automatically sails to the target observation area according to the destination position information of this observation task by relying on the GPS navigation system.

[0174] The sail unmanned boat is an unmanned ocean observation load-carrying platform that can sail automatically and execute observation tasks automatically. It has the advantages of long endurance, low power consumption, and a wide coverage area of the observation range. Using the sail unmanned boat to observe the Langmuir circulation in a large area can obtain continuous long-term observation data in terms of time and space, which will greatly enrich the existing Langmuir circulation observation data, facilitate the research on the Langmuir circulation, and thus provide technical support for further marine oil spill treatment and maritime personnel search and rescue.

[0175] The hull of the sail unmanned boat is designed with a smooth-profile monohull structure. The smooth and closed outer contour curve is beneficial to reducing the vibration impact on the hull when the wind speed increases and the wave conditions are harsh. At the same time, the smooth outer contour curve reduces the resistance suffered by the ship during navigation, saves the consumed energy, and enables the ship to execute the Langmuir circulation observation task for a longer time under the same load conditions.

[0176] The sail part of the sail unmanned boat adopts a wing sail structure. Considering that the wing sail structure belongs to a balanced sail structure, the wing sail rotating shaft is arranged at the aerodynamic center to reduce the force required to rotate the wing sail, and a low-power motor is further used to drive the wing sail to rotate.

[0177] Solar panels are inlaid on the surface of the wing sail, which can charge the on-board devices and charge the battery when the power demand of the observation equipment is met. The battery can provide power for the propeller to drive the ship forward. The sail unmanned boat adopts three driving modes, namely: wind power driving mode, propeller driving mode, and hybrid power mode. The hybrid power mode includes sail driving and propeller driving. When the wind condition is good and the wind power is sufficient to drive the ship forward, the sail unmanned boat automatically raises the foldable wing sail and adopts the wind power driving mode to drive the unmanned boat forward with the wind energy. When the wind condition is poor, the wind power is small, and the ship is sailing against the wind, the ship cannot obtain sufficient sailing speed, so the foldable wing sail is folded up and the propeller driving mode with the propeller propelled by the battery power is adopted.

[0178] The sail unmanned boat uses its own GPS satellite navigation system to determine its own position. It uses a wireless transmission system to communicate with the remote control center to determine the target area position of the observation task. The wireless transmission system includes an industrial router, which is used to transmit the position information, navigation information and some data information collected by the sail unmanned boat to the remote control center, so as to control the behavior of the sail unmanned boat. At the same time, through the wireless transmission system, the sail unmanned boat can receive the longitude and latitude coordinates of the target task area and the longitude and latitude information of the boundary of the target area sent by the remote control center.

[0179] The sail unmanned boat relies on the main control system to determine the sailing path and behavior logic of the sail unmanned boat according to the heading angle, six-axis attitude information, wind direction information of the sail unmanned boat obtained by the sensor, and the longitude and latitude information of the ship and the longitude and latitude information of the target area received by the GPS system. Specifically, the sail unmanned boat is designed with two different sailing control modes: long-range mode and operation mode. The long-range mode is applicable when the unmanned sailboat is on the way to the observation area. In this mode, the main control system selects the correct path to the target observation area according to the wind direction, ship coordinates, target point and other information received by the sensor, and determines the position of the next sailing point through local path planning. Specifically, the branch scenario tree and potential field method are used for the dynamic path planning of the unmanned sailboat. At the same time, the main control system uses the interval analysis method to avoid static obstacles during the process of going to the path point, and uses the combination of thermal imaging and radar technology to identify dynamic obstacles, so as to avoid dynamic obstacles on the driving path.

[0180] When the sail unmanned boat arrives at the target observation area, the main control system controls the sail unmanned boat to enter the operation mode. Specifically, there are three different modes for the path planning strategy of the sail unmanned boat in the operation mode, namely, the right-angle path along the edge, the vertical bubble mark path, and the sailing path along the dyed strip.

[0181] In the second step, when the sail unmanned boat reaches the target observation area, a dye tracer is automatically released into the water body.

[0182] When the sail unmanned boat sails to the Langmuir circulation generation area, the sail unmanned boat automatically releases a pollution-free and degradable dye tracer into the water body, so that the dye spreads with the water body exchange process of the Langmuir circulation, thereby dyeing the water body. Specifically: when the sail unmanned boat reaches the observation area, multiple quadrotor unmanned aerial vehicles are dispatched to collect images of the environment around the unmanned boat from the air. First, a drone is dispatched from the position where the unmanned boat is located. The drone flies straight up into the air and takes a water surface image. When the number of bubble marks n captured by the drone is ≥ 3, the drone hovers at this height. Then, the sail unmanned boat reaches the middle position between two strip-shaped bubble marks under the guidance of the drone. Since the positions of the two bubble marks are in the upwelling section of the Langmuir circulation, and the area between the bubble marks belongs to the downward jet section of the Langmuir circulation, the downward jet section has a downward jet flow velocity of up to 20 cm / s. Releasing the dye in this section can promote the rapid diffusion of the dye in the entire Langmuir circulation observation area. The unmanned boat releases the dye tracer at water depths of 0, 1, 2, and 3 m at intervals of 1 m from the water surface to 3 m underwater in this area to dye the water body. The water body dye used is a water-soluble β-carotene dye, which can dye the water body orange-yellow, making it easier to distinguish and observe the circulation dynamics of the water body. Moreover, this dye is one of the carotenoids, the most common and stable natural pigment in nature. It belongs to natural chemicals. It exists abundantly in plants, giving fruits and vegetables their rich yellow and orange colors. Therefore, its impact on the environment is relatively small. After the dyeing of this area is completed, under the guidance of the drone, the sail unmanned boat goes to the next bubble mark generation area for dyeing. The next area is located outside the left and right edge bubble marks. After arriving at the next area, repeat the above process to dye the water body until the last bubble mark strip.

[0183] In the third step, after the dye tracer is released, a quadrotor unmanned aerial vehicle is dispatched. The quadrotor unmanned aerial vehicle starts from the location of the unmanned boat and follows the diffusion path of the dye to collect images.

[0184] When the sail unmanned boat releases the water body dye and it spreads sufficiently, the on-board drone Langmuir circulation bubble mark tracking system is used to track the dyed bubble marks. The specific composition of the drone Langmuir circulation bubble mark tracking system is as Figure 3 shown.

[0185] The composition of the entire system can be roughly divided into four parts, namely the unmanned aerial vehicle (UAV) aerial platform, the bubble mark tracking and detection system, the boat-aircraft transmission system, and the unmanned boat data processing platform. The UAV aerial platform is equipped with a microcomputer flight control system responsible for controlling the flight attitude of the UAV; the on-board D-RTK can provide centimeter-level elevation and planar positioning accuracy, and information is transmitted between the UAV and the unmanned boat through the boat-aircraft transmission system; the gimbal camera can stably capture 20 million-pixel photos and high-definition 60fps videos. Among them, the sail unmanned boat is equipped with a dual-antenna GPS receiver, which receives GPS signals to provide stable attitude, position, speed, heading, clock and other data for the mission equipment. Commands and data are sent and received between the unmanned boat and the UAV through 2.4GHz real-time radio frequency point-to-point communication. The video data collected by the UAV is transmitted and recorded in real time in the storage device on the unmanned boat. The unmanned boat data processing platform analyzes the image data transmitted back by the UAV, makes a flight path plan for the UAV, and controls the UAV to fly and observe according to the established route through the boat-aircraft transmission system. The specific steps are as described in the sixth step.

[0186] In the fourth step, use image recognition technology to recognize the collected images and determine whether they are Langmuir circulation.

[0187] Since using deep learning methods to detect objects requires a relatively high depth for the network model, a very large depth, and the number of parameters in the network reaches millions. This requires the network to use a large amount of training data during training. If a large amount of data is not used to train the network, the training results will be affected and overfitting will occur. Figure 4 It is a general flowchart for image recognition. Therefore, the prerequisite for image recognition is to obtain a large number of images of Langmuir circulation bubble marks. The image sources used to train the classifier in this application are Langmuir circulation bubble marks on the Internet, strip-shaped marks formed by the diffusion of oil spills at sea as the circulation spreads, images collected after each execution of the circulation detection task, and image frames obtained by splitting the videos taken by the UAV. As Figure 5 shown in the Langmuir circulation recognition flowchart. Then, since the total number of collected images is small, they are divided into a training set, a validation set, and a test set by stratified random sampling according to a ratio of 6:2:2.

[0188] After the dataset is partitioned, preprocess the training set data. The preprocessing process includes image annotation and data augmentation. Image annotation is performed using Labelimg. Labeimg is an open-source and free image annotation tool written in Python. Open Labeimg with the anaconda terminal and manually annotate the collected Langmuir circulation bubble marks. The annotation uses a rectangular box to select the Langmuir circulation bubble marks or the strip marks formed by the diffusion of oil spills in the image, and finally generates an xml format file for storage. As Figure 6 shown in the annotation information of the Langmuir circulation bubble marks, including the overall information of the original image and the annotation box information. The overall information of the original image includes the width, height, and number of channels of the image; the annotation information includes the maximum and minimum coordinates of the horizontal and vertical axes of the rectangular box.

[0189] Since the actual data available is not as sufficient as desired, there is no publicly available dataset for the images of Langmuir circulation bubble marks. To avoid overfitting during model training and conduct better training of the neural network, it is necessary to process the data images, expand the number of images, and perform data augmentation. Therefore, for the above-mentioned annotated Langmuir circulation image training set data, the following six data augmentation methods are used to expand the images: (1) Data flipping, which is like mirroring the image rather than rotating it 180 degrees; (2) Data rotation, which rotates the existing image data clockwise or counterclockwise by a certain angle to obtain new image data; (3) Data scaling. Using this method, the image data can be enlarged or reduced. After enlargement, the part exceeding the original image size will be cropped, and after reduction, the part between the boundary and the original size boundary will be assumed to be filled; (4) Data cropping, which randomly selects a part of the image and then cuts off the area outside the selected part and adjusts the selected part to the original image size; (5) Data translation, which randomly moves the target object in the image in two directions; (6) Adding noise, which can eliminate the high-frequency features in the image and reduce the occurrence of overfitting. The bubble mark position annotation work is carried out before the image data augmentation operation to avoid the small errors caused by the scaling of the annotation box. Moreover, the annotation work is only for the original dataset, and the geometric transformation, histogram equalization, and noise addition processing of the image will not affect the position annotation information. For the geometric transformation of the image, only need to parse its annotation information and modify the coordinate information according to the transformation angle and method to obtain the new annotation information.

[0190] Next, use the YOLOv5 algorithm based on transfer learning to perform deep learning on the Langmuir circulation bubble mark training set images, so as to achieve image recognition and positioning of Langmuir circulation bubble marks. The YOLOv5 object detection algorithm is an improvement of the YOLOv3 algorithm. Improvements have been made in aspects such as network structure and training techniques on the basis of the YOLOv3 algorithm, resulting in further improvement of the detection performance. The framework of YOLO series object detection can usually be divided into the following parts: input end, backbone network, Neck network, and output end. The YOLO method solves object detection as a regression problem and completes the output from the input of the original image to the object position and category based on a single end-to-end network. While rcnn, fast rcnn, and faster rcnn solve the detection results in two parts: object category (classification problem) and object position (regression problem). Transfer learning is a type of machine learning that applies the knowledge and skills learned in a certain field or task to a new field or task. In deep learning, the training of network models has extremely high hardware requirements for computers, consumes a large amount of time and computing resources during the training process, and the same is true during the parameter tuning process, and sometimes the expected effect cannot even be achieved. Using transfer learning can greatly reduce the operation time and computer resources. The important weights in the already trained model can be saved and then transferred to the new network model. In this way, both the low-level features of the image can be obtained, and the use of time and computer resources is also reduced. Sometimes in a new task field, the obtained sample data set may not be particularly sufficient. In order to make the model achieve better performance, training an insufficient sample data set may lead to model overfitting. By using the transfer learning method, the relatively mature model parameters in a similar field can be transferred to the new task, and the overfitting problem caused by insufficient sample data set can be solved by learning the already trained low-level features. Therefore, in this application, knowledge transfer is performed on the object detection model pre-trained on the coco data set on YOLOv5, and the Langmuir circulation bubble mark image data set is trained to obtain the YOLOv5 Langmuir circulation bubble mark recognition model. Then, knowledge transfer is performed on the YOLOv5 Langmuir circulation bubble mark recognition model to train the Langmuir circulation bubble mark recognition model after staining, such as Figure 7 is the multiple transfer learning process for training the YOLOv5 Langmuir circulation bubble mark recognition model.

[0191] Next, the recognition model is used to perform real-time recognition on the dyed Langmuir circulation bubble marks. The data processing platform of the unmanned boat first classifies the dyed red water body and the undyed blue-green water body. First, a data preprocessing process is carried out. Since the external environment during the image acquisition by the unmanned aerial vehicle changes in real time and is uncertain, there will inevitably be color deviations in the acquired images. Therefore, the acquired images are first subjected to color restoration. The Spydercheckr 24-color standard color card of Datacolor company is used for correction. By extracting the relationship between the color values of the color blocks in the color card in the image and the standard values of the color card, the color restoration matrix is determined to perform color restoration on the image. Because the shooting condition is under natural light, the color difference caused by the sensor itself is not very large. Therefore, the color restoration matrix should be close to the identity matrix, and the values of each element should be kept near 1.0 or 0.0. Therefore, an iterative method is used to generate a set of color restoration matrices Sc.

[0192] Such as a 11 = 1.0 + 0.02x; a 12 = 0.0 + 0.02x; a 13 = 0.0 + 0.02x,

[0193] where x = 0, 1, 2, …, 24, 25.

[0194] Thus, for the vector V R =(a 11 , a 12 , a 13 ) there are 26 3 possible values. Similarly, V G , V B can be iterated. For V R , V G , V B (V R , V G , V B represent the values of the RGB color components respectively), any one of their values can be combined into a color restoration matrix C, and all the combined matrices constitute the set of color restoration matrices Sc. The variance is calculated between all possible values and the standard values corresponding to each color block

[0195]

[0196] The matrix corresponding to the minimum variance is used as the optimal color restoration matrix C. Finally, this color restoration matrix is used to correct the color deviation of the image, so that the image color is not affected by environmental factors such as light and weather, which is convenient for subsequent classification and recognition. The formula for color deviation correction is as follows

[0197]

[0198] Where C is the color restoration matrix, R, G, and B are the color values in the original image, and R’, G’, and B’ are the color values of the image after color cast correction. Next, use the YOLOv5 algorithm model to complete the feature extraction of the image to be measured, obtaining the total number M of pictures with bubble marks, the total number N of bubble mark areas, and the average time T required to identify each of the 100 pictures, and calculate the accuracy P of bubble mark identification and the accuracy P of bubble mark area identification 2 . When the accuracies are both greater than 90%, it is determined that there are Langmuir circulation bubble marks in this area.

[0199] Step 5: When it is determined to be Langmuir circulation, set the data collection route for the unmanned boat according to the collected images. Send out multiple unmanned boats to jointly detect this area.

[0200] Process the captured images to identify the edges of the bubble marks to facilitate the setting of the observation path. Such as Figure 8As shown, in this embodiment, the collected color JEPG format image is first grayscaled using the maximum value method. The maximum value among the R, G, and B components of the image is taken as the grayscale value, and the grayscaled image is displayed according to the grayscale value. Next, the mean filter algorithm is used to perform mean filtering on the grayscaled image. The center point and the four corners of the image are set as the points to be processed. The fixed step size in the direction is set to one percent of the pixel value of the image, and the filter window is set to one-thousandth. The sliding process is performed on the image to be measured. Next, when the filter window slides to the pixel point (x, y), the average value of the neighboring pixel points of this point is calculated, and the value of this pixel point is replaced with the average value. Next, the above steps are repeated to complete image denoising, and the denoised grayscale image is obtained. Next, global binarization processing is performed on the image, and the function cv2.threshhold() in the OpenCV graphics library is used for global binarization. Next, canny edge detection is performed on the image that has been denoised, grayscaled, and binarized. The specific steps are as follows: (1) Perform convolution calculation through a Gaussian filter to achieve the function of noise reduction; (2) Obtain the gradient through the Sobel operator; (3) Non-maximum suppression to screen out other elements that do not belong to the edge; (4) Double-threshold screening: Two thresholds are default set in the canny algorithm, and they are divided into high threshold and low threshold according to the threshold size; (5) Hysteresis boundary tracking. For each weak edge point, its corresponding 8-connected neighborhood pixels are checked. If there is a strong edge point, then this weak edge point is generated by the real edge, and thus the weak edge points caused by noise are excluded, and the weak edge points generated by the real edge points are retained. Next, the cv2.findContours() function in OpenCV is used to detect the edge of the bubble trace. The parameters of this function are set as (1) mode: cv2.RETR_EXTERNAL: Detect the outer edge of the stained bubble trace; (2) cv2.CHAIN_APPROX_NONE: Used to obtain the positioning of the bubble trace edge.

[0201] There are three different modes for the unmanned boat to observe, namely the right-angle path along the edge, the path perpendicular to the bubble trace, and the path along the stained strip.

[0202] Among them, the flowchart for generating the right-angle path along the edge is as Figure 9 shown:

[0203] First, search for and cluster the convex hull points of the graph, and then determine the target convex hull points. Assume there are N convex hull points, and there are N - 1 intervals between the convex hull points. Set a partition reference distance d, and record the total number of intervals greater than the reference distance as C. When C is greater than the number of unmanned boats n, the convex hull points are divided into C categories; when C is less than n, since n unmanned boats are selected and each unmanned boat must select a convex hull point as the target convex hull point, the convex hull points are divided into n categories. Next, take the position of the unmanned boat as the initial point, draw a circle with the line connecting the initial point and the target convex hull point as the diameter, and use this line segment as the offset line segment; then, take the initial point as the fixed point, offset in a certain angle in the direction away from the bubble mark area, find the first line segment that does not cross the bubble mark area after offset, and define the intersection point of this line segment and the circle as the target circle point; then, connect the initial point and the target convex hull point to the target circle point respectively, and a path from the initial point to the target point is obtained, and the included angle between the two line segments forming this path is a right angle; similarly, take the target convex hull point as the fixed point, operate using the above method and find the target circle point, and a path from the target point to the initial point is obtained; finally, connect the intersection points of the two paths to the initial point and the target convex hull point respectively, and the final path of the unmanned boat observation is obtained.

[0204] Among them, the flowchart for generating the vertical bubble mark path is as Figure 10 shown:

[0205] First, approximate the zigzag bubble mark as a straight line strip with a width. Next, determine the path of the unmanned boat according to the relative positions of the two approximately straight-line bubble marks. Specifically: when the two bubble marks are parallel, the path of the unmanned boat adopts a path perpendicular to its starting bubble mark; when the two bubble marks are not balanced and there is an included angle in the distance, take the position of the intersection point of the two bubble marks as the center of the circle, and use concentric arcs between the two bubble marks as the observation path of the unmanned boat. The observation path of the unmanned boat is planned in real time according to the real-time observation of the unmanned aerial vehicle. When the position of the bubble mark changes, optimize the path of the unmanned boat to achieve the purpose of making the detection path of the unmanned boat the longest between the two bubble marks.

[0206] Among them, the flowchart for generating the path along the dyeing strip is as Figure 11 shown:

[0207] Specifically, a video acquisition module is installed on the unmanned boat. The video acquisition module uses a low-resolution black-and-white camera combined with a single-chip microcomputer to identify and process images. The entire video acquisition module is characterized by its ability to actively search for dye marks, enabling the camera to actively adjust its parameters according to the distance and position between the unmanned boat and the dye strip marks, and collect dye mark images under different postures and different lens imaging parameter conditions. The entire platform includes a servo motor bracket, a camera bracket, and a servo motor. The specific process of tracking the sailing path of the dye strip is as follows: The camera continuously changes as the position of the dye strip changes, ensuring that the position of the dye strip is always in the middle of the camera's captured image. The rotation of the camera is controlled by the servo motor at the bottom of the camera bracket, and the control method uses fuzzy control rules. The fuzzy control rules define an input k, where k represents the relative position of the dye strip with respect to the unmanned boat in the unmanned boat coordinate system. Five fuzzy linguistic variables of k are defined: left (LB), slightly left (LS), zero (O), slightly right (RS), and right (RB). By controlling the camera position to ensure that the dye strip is in the center of the captured image, the sailing direction of the sail unmanned boat is controlled according to the rotation angle of the servo motor. For example, when the servo motor rotates to the left, the sail unmanned boat turns to the left along with the rotation direction of the servo motor, gradually making the rotation angle of the servo motor zero, and then the servo motor rotates again according to the position of the dye strip. Repeating the above steps ensures that the sailing path of the small boat always follows the dye strip.

[0208] Step 6: The sail unmanned boat conducts a cruising observation along the set route, and at the same time, a drone swarm is dispatched to collect images of the entire Langmuir circulation area from the air.

[0209] When the sail unmanned boat reaches the predetermined position, the sail unmanned boat releases the drones. After all the sail unmanned boats in the entire observation area have released the drones, the drones start to form an observation network. The communication network of the drone swarm adopts a hierarchical distributed networking method, organizing all the drones in this area into hierarchical node groups, and at the same time designating any one of the drones as the group routing node of this node group, responsible for the information transmission between this hierarchical node group and other hierarchical node groups. The network structure diagram is as Figure 12 shown:

[0210] As the bubble marks of the Langmuir circulation continue to move and change, the positions of the nodes in the unmanned aircraft networking system are also constantly moving and changing. Therefore, the established communication paths will also change continuously. If there are multiple available communication paths between the source node and the destination node in the network, there will be a problem of path selection. Through the active network, an appropriate routing can be selected to obtain better network characteristics. In Figure 12The dotted arrow in the figure indicates the communication path that the network chooses to establish after the original communication path is destroyed. After the drone is launched into the air and networked, it forms a complete three-level network system architecture with the surface unmanned boat. The first level is the unmanned boat data processing platform, which is equipped with computing equipment and positioning equipment, responsible for data processing, command, and providing image control point measurement services for drone aerial photogrammetry and image acquisition; the second level is the drone platform, equipped with RTK and pan-tilt photography equipment, etc., responsible for image acquisition and photogrammetry; the third level is the underlying data acquisition equipment. The physical link between the first and second levels uses wireless network (3G / 4G, wireless image transmission); the physical link between the second and third levels uses wired direct connection (serial port, USB, HDMI and other equipment with their own protocols); the bottom layer collects the original image data, and later performs data imaging splicing, information extraction and analysis.

[0211] After forming a drone swarm to collect aerial images, the full-area image of the Langmuir circulation generation area is obtained by stitching and fusing the images of all the drones, and then the area size of the Langmuir circulation generation area is estimated through geometric relationships.

[0212] (1) First, it is necessary to establish a UAV detection model to determine the detection range and the corresponding geometric position relationship. In the Langmuir circulation observation area E, there are N t Langmuir circulation bubble trace {T i ,i=1,2,…,N t} and N v A drone with the same structure {V i ,i=1,2,…,N v All drones have the same configuration and performance and can fly above the sea at a height h. The flight height h is limited by the performance of the drone and h∈[h min ,h max ], discretize the flight altitude interval into λ discrete altitudes h λ (λ=1,2,…,n), in one decision cycle, all drones are at the same height h λ Down with speed v λ In the inertial reference coordinate system, its motion model is:

[0213]

[0214] In formula (1): is the position of the i-th UAV in the sea space; is the yaw angle; u i ∈[-1,1] is the decision variable, and u(k)=[ui(k),i=1,2,…,Nv], u(k) is the set of decision variables of all UAVs at time k; η max is the maximum turning angular velocity of the UAV. is the state vector, and the set of state vectors of all UAVs is s(k) = [s i (k), i = 1, 2, …, N v .

[0215] Next, an image sensor model is established. A pan-tilt camera is installed under each UAV. During the actual detection process of Langmuir circulation, the quality of the captured images is affected by the uncertainty of the camera's working state and the uncertainty of environmental changes, resulting in changes in image quality, resolution, etc. An image sensor model considering altitude changes is established as Figure 13 shown.

[0216] Among them, the rectangular box represents the observation range of the UAV. The size of the sea area detected by the UAV is expressed as E λ = λ × w λ , where l λ and w λ are the length and width of the effective picture instantaneously detected by the sensor at altitude h λ respectively, and the calculation method is as follows.

[0217]

[0218] In Equation (2): β s and γ s are the field of view angles of the sensor respectively.

[0219] After establishing models for UAVs and image sensors, since the UAV swarm and the surface unmanned boat are interconnected through hierarchical distributed networking, after the unmanned boat obtains the image data and the set of UAV state vectors sent by the UAV swarm, according to the principles of aerial photogrammetry, it judges whether the overlap degree of the sea area detection between two adjacent UAVs is not less than 40% horizontally and not less than 60% vertically. If it is lower than this threshold, according to the image sensor model, combined with the UAV camera parameters and the set of UAV state vectors, the detection range of the sensor at this time is estimated. Then, by adjusting the UAV flight altitude or the positions of UAVs relative to each other, the overlap degree in the image sensor model is made to meet the threshold requirements. After the adjusted parameters meet the threshold requirements, the surface unmanned boat transmits the generated set of decision variables to the UAVs through hierarchical distributed networking to control the UAVs to adjust their flight altitude and positions.

[0220] (2) After the UAV swarm obtains the full-area image of the Langmuir circulation generation area, it first performs image registration and then image fusion to obtain a regional panoramic image.

[0221] Image registration first detects feature points, that is, control points in the image. In this application, an unmanned surface vehicle is selected as the image feature points of an area, and the unmanned surface vehicle measures its accurate position information through RTK, so that the geographical location information of the registered image is accurate. At the same time, the SIFT feature point detection algorithm is used to detect feature information such as the edge of the bubble mark, the edge of the staining strip, and the middle band between two bubble marks. The SIFT algorithm identifies feature points by finding local extrema in the scale space of the image. Then, the gradient histogram is calculated for the pixel points within a certain neighborhood of each extreme point. Finally, a feature vector consisting of 128 data is generated through the gradient histogram to describe the feature points.

[0222] Next, after extracting the SIFT feature descriptors, it is necessary to extract the correct matching feature points in the reference image and the image to be registered. First, the Euclidean distance of the descriptors is used to determine whether the feature points match. Then, the following two steps are used to screen the feature matching point pairs to eliminate the false matching of the feature points and ensure the accuracy of the key point matching. ① A matching strategy of comparing the ratio of the Euclidean distance between a certain feature point E and its nearest neighbor point P and the second nearest neighbor point P' is used for screening. ② Use the random sample consensus to further delete the wrong matching point pairs. Calculate the homography between the two images and find the outliers, that is, the incorrect matching points, to improve the matching rate of the image feature points.

[0223] After obtaining the accurate matching feature point pairs of the two images, the homography matrix H can be solved according to the image transformation relationship of formula (3). Through H, the image I' to be registered can be transformed into the same coordinate system as the reference image I.

[0224]

[0225] In the formula: (x′, y′) and (x, y) are the matching point pairs of the image I' to be registered and the reference image I.

[0226] (3) After converting the image I' to be registered and the reference image I into the same coordinate system, the direct image fusion algorithm is adopted, that is, the overlapping area on a single image is taken for image fusion. In order to overcome the cumulative error generated when stitching multiple images, the image with the unmanned surface vehicle as the control point is selected as the best reference image, and a method of multi-image stitching through grouped stitching is adopted here. That is, first stitch the images within the group, and then stitch the images between the groups. During the stitching process of the images between the groups, the registration parameters of the adjacent two groups of images will be recalculated, and the image transformation will be performed in groups, so as to reduce the number of image transformations and further reduce the accumulation of stitching errors. In this application, on the premise of ensuring a 40% image overlap rate, a grouped image stitching method with 4 images in a group is selected to solve the problem of easy stitching errors in multi-image stitching. The process of stitching grouped images is as Figure 14as shown

[0227] (4) After obtaining the panoramic image of the Langmuir circulation generation area, estimate the approximate range of the Langmuir circulation generation area through geometric relationships. First, use the aforementioned edge detection method to determine the circulation generation edges in the full-area image, then estimate the scaling ratio of the image based on the position and size scale of the unmanned boat measurement and control points, and further calculate the influence range of the Langmuir circulation generation area.

[0228] In summary, the observation process of the present invention is flexible, low-cost and highly efficient.

[0229] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order mentioned in the operation diagrams. For example, depending on the functions / operations involved, two consecutive blocks shown may actually be executed substantially simultaneously or the blocks can sometimes be executed in the reverse order. In addition, the embodiments presented and described in the flowcharts of the present invention are provided by way of example for the purpose of providing a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated where the order of various operations is changed and where sub-operations described as part of a larger operation are executed independently.

[0230] In addition, although the present invention has been described in the context of functional modules, it should be understood that unless otherwise stated to the contrary, one or more of the functions and / or features described may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It should also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. Rather, considering the attributes, functions and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the modules will be understood within the ordinary skills of an engineer. Therefore, those skilled in the art can implement the present invention as set forth in the claims without undue experimentation. It should also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0231] If the above-described functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0232] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in conjunction with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0233] More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer diskette case (magnetic device), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber device, and portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other appropriate processing if necessary, and then storing it in a computer memory.

[0234] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one of the following techniques known in the art or a combination thereof can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0235] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0236] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the claims and their equivalents.

[0237] The above has specifically described the preferred embodiments of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. An observation control method for Langmuir circulation, characterized in that, it includes: Configuring an unmanned boat to a target observation area according to the destination location information of the target observation task; Releasing a dye tracer into the water body through the unmanned boat; Taking the location of the unmanned boat as the starting point, configuring an unmanned aerial vehicle to follow the diffusion path of the dye tracer for image acquisition; Performing image recognition on the acquired images to determine whether it is a Langmuir circulation; When a Langmuir circulation is recognized, configuring a data acquisition route for the unmanned boat according to the acquired images, controlling the unmanned boat formation to detect the Langmuir circulation area according to the data acquisition route, and performing image acquisition and recognition on the Langmuir circulation area through an unmanned aerial vehicle cluster to complete the edge detection of the Langmuir circulation area; The performing image recognition on the acquired images to determine whether it is a Langmuir circulation includes: Obtaining training images to be trained, where the training images to be trained include Langmuir circulation bubble marks in the Internet, strip-shaped marks formed by the diffusion of oil spills at sea following the circulation, images acquired after each execution of the circulation detection task, and image frames obtained by splitting frames from videos taken by unmanned aerial vehicles; Dividing the training images to be trained into a training set, a validation set, and a test set by means of stratified random sampling; Performing image annotation processing and data augmentation processing on the training set to obtain training set images; Using the YOLOv5 algorithm based on transfer learning to perform deep learning on the training set images to construct a Langmuir circulation bubble mark recognition model; Performing image recognition on the acquired images according to the Langmuir circulation bubble mark recognition model to determine whether it is a Langmuir circulation; Among them, the performing data augmentation processing on the training set includes at least one of the following: Performing data flipping on the training set to obtain the first data; Or, performing data rotation on the training set to obtain the second data; Or, performing data scaling on the training set to obtain the third data; Or, performing data cropping on the training set to obtain the fourth data; Or, performing data translation on the training set to obtain the fifth data; Or, performing noise addition processing on the training set to obtain the sixth data; The using the YOLOv5 algorithm based on transfer learning to perform deep learning on the training set images to construct a Langmuir circulation bubble mark recognition model includes: Performing knowledge transfer on the object detection model pre-trained on the YOLOv5 for the coco dataset, training the Langmuir circulation bubble mark image dataset to obtain the YOLOv5 Langmuir circulation bubble mark recognition model; Then performing knowledge transfer on the YOLOv5 Langmuir circulation bubble mark recognition model to train the Langmuir circulation bubble mark recognition model after dyeing; The performing image recognition on the acquired images according to the Langmuir circulation bubble mark recognition model to determine whether it is a Langmuir circulation includes: Performing color restoration processing on the acquired images to generate a set of color restoration matrices; Performing color deviation correction and restoration on the acquired images according to the set of color restoration matrices; Use the Langmuir circulation bubble mark recognition model to perform real-time recognition on the restored image, classify and determine the dyed red water body and the undyed blue-green water body, and then realize the recognition of the Langmuir circulation.

2. An observation and control method for Langmuir circulation according to claim 1, characterized in that, the releasing of the dye tracer into the water body by the unmanned boat includes: When the unmanned boat reaches the target observation area, dispatch a drone from the position where the unmanned boat is located. The drone flies straight up into the air and takes a water surface image; When the number of bubble marks captured by the drone is greater than or equal to the first threshold, hover the drone at the current altitude; Under the guidance of the drone, control the unmanned boat to reach the middle position between two strip-shaped bubble marks, and the unmanned boat releases the dye tracer from the water surface and underwater at the current position.

3. An observation and control method for Langmuir circulation according to claim 1, characterized in that, configuring a drone to follow the diffusion path of the dye tracer to collect images starting from the location of the unmanned boat, includes: Controlling the drone to perform aerial operations through the drone aerial platform and taking images; Transmitting the images captured by the drone through the boat-aircraft transmission system and storing them on the unmanned boat; After processing the stored images through the unmanned boat data processing platform, plan the flight path of the drone and send it to the drone aerial platform to control the drone to further obtain images according to the flight path.

4. An observation and control method for Langmuir circulation according to claim 1, characterized in that, when the Langmuir circulation is recognized, configure the data collection route of the unmanned boat according to the collected images, control the unmanned boat formation to detect the Langmuir circulation area according to the data collection route, and perform image collection and recognition on the Langmuir circulation area through a drone cluster to complete the edge detection of the Langmuir circulation area, including: Performing grayscale processing on the collected images by the maximum value method to obtain grayscale images; Performing mean filtering on the grayscale images through the mean filtering algorithm to complete the sliding processing of the grayscale images; Slide the filtering window to the target pixel point, calculate the average value of the neighboring pixel points of the pixel point, and replace the value of the pixel point with the average value of the neighboring pixel points until the denoising processing of all pixel points is completed; Perform global binarization processing on the denoised grayscale images and then perform canny edge detection to detect the edges of the bubble marks and complete the edge detection of the Langmuir circulation area.

5. An observation and control method for Langmuir circulation according to any one of claims 1-4, characterized in that, there are three observation modes for the unmanned boat, including the mode of observing along the right-angle path of the edge, the mode of observing along the path perpendicular to the bubble marks, and the mode of observing along the navigation path of the dye strip; Among them, the mode of observing along the right-angle path of the edge includes the following steps: Search for and cluster the graphic convex hull points to determine the target convex hull points; Set a partition reference distance, and determine the category of the target convex hull points according to the number of partitions greater than the partition reference distance; Determine the position of the offset line segment based on the position of the unmanned boat; Find the first line segment that does not cross the bubble trace area after offset, and define the intersection point of this line segment and the circle as the target circle point; Connect the initial point and the target convex hull point to the target circle point respectively, to obtain a first path from the initial point to the target point, and the included angle between the two line segments forming the first path is a right angle; Find the target circle point and determine a second path from the target point to the initial point; Connect the intersection points of the first path and the second path to the initial point and the target convex hull point respectively, then the final path observed by the unmanned boat is obtained; The mode of observing along the vertical bubble trace path includes the following steps: The UAV provides real-time images of a large range of bubble traces in the air, and uses edge detection to detect the edges of the bubble traces; Approximate the zigzag bubble trace as a straight line strip with a width; Determine the path of the unmanned boat according to the relative positions of the two approximately straight-line strip bubble traces; among them, when the two bubble traces are parallel, the path of the unmanned boat adopts a path perpendicular to the starting bubble trace; when the two bubble traces are unbalanced and there is an included angle in the distance, use the position of the intersection point of the two bubble traces as the center of the circle, and use concentric arcs between the two bubble traces as the observation path of the unmanned boat; Perform real-time planning on the observation path of the unmanned boat according to the real-time observation of the UAV, and optimize the path of the unmanned boat when the position of the bubble trace changes; The mode of observing along the navigation path of the dyeing strip includes the following steps: Use the video acquisition module on the unmanned boat to track the dyeing trace, actively adjust the camera parameters and posture, and perform image acquisition; Use fuzzy control rules to control the rotation of the servo steering gear at the bottom of the camera bracket; The camera continuously changes as the position of the dyeing strip changes, so that the trace is located in the center of the image.

6. A method for observing and controlling Langmuir circulation according to any one of claims 1-4, characterized in that, The method further includes the steps of stitching and fusing the images of all UAVs to obtain a full-area image of the Langmuir circulation generation area, and then estimating the area size of the Langmuir circulation generation area through geometric relationships. The steps specifically include: Establish a UAV detection model to determine the detection range and corresponding geometric position relationships; Establish an image sensor model; According to the UAV detection model and the image sensor model, after obtaining the full-area image of the Langmuir circulation generation area, first perform image registration, and then perform image fusion to obtain a regional panoramic view; After obtaining the regional panoramic view of the Langmuir circulation generation area, estimate the range of the Langmuir circulation generation area through geometric relationships.

7. A system for implementing the method for observing and controlling Langmuir circulation according to any one of claims 1-6, characterized in that, including: The first module is used to configure the unmanned boat to the target observation area according to the destination position information of the target observation task; The second module is used to release a dye tracer into the water body through the unmanned boat; The third module is used to start from the location of the unmanned boat and configure the UAV to follow the diffusion path of the dye tracer for image acquisition; The fourth module is used to perform image recognition on the collected images to determine whether it is Langmuir circulation; The fifth module is used to, when the Langmuir circulation is recognized, configure the data acquisition route of the unmanned boat according to the collected images, control the unmanned boat formation to detect the Langmuir circulation area according to the data acquisition route, and perform image acquisition and recognition on the Langmuir circulation area through the UAV cluster, so as to complete the edge detection of the Langmuir circulation area.

Images