Method and apparatus for detecting water sports object based on unmanned aerial vehicle, electronic device

Abstract

The application provides a kind of detection method, device and electronic equipment based on unmanned aerial vehicle for water sports object, which comprises obtaining visible light image, infrared image and radar image by using unmanned aerial vehicle;Visible light image, infrared image and radar image are fused to obtain three-dimensional scene model;Based on three-dimensional scene model, water body boundary contour is extracted, and water area range is determined according to water body boundary contour;The geometric features and texture patterns of the preset water target are obtained, and whether the preset water target exists in the water area range is detected based on the preset target detection algorithm;When the preset water target exists in the water area range, the flight of the unmanned aerial vehicle is controlled according to the motion state of the preset water target.It can better adapt to environmental variables, and can make real-time decisions according to the motion of the target, effectively track, and thus make the detection of water target more accurate.

Description

Methods, devices, and electronic equipment for detecting moving objects on water based on unmanned aerial vehicles (UAVs). Technical Field

[0001] This invention relates to the field of detection technology for moving objects on water, and particularly to a method, device, and electronic device for detecting moving objects on water based on unmanned aerial vehicles (UAVs). Background Technology

[0002] In recent years, the detection of moving objects on water has become increasingly important, for example, given the prevalence of smuggling activities and the presence of smuggling vessels on the water. Different solutions have been proposed in existing technologies for detecting these moving objects. One approach incorporates partially automated tools, such as analysis software based on satellite remote sensing imagery, but its core relies primarily on static deployment or low-frequency manual patrols. Another approach uses multi-rotor drones for assisted detection, but these drone patrols employ pre-planned routes, such as parallel scanning or circular encirclement, using simple geometric paths and lacking dynamic adaptive adjustment mechanisms. Current technological solutions all rely on historical experience data to set patrol ranges and route parameters, failing to fully integrate real-time environmental variables and target behavior characteristics to achieve intelligent decision-making. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, device, and electronic device for detecting moving objects on water based on unmanned aerial vehicles (UAVs) to address the aforementioned technical problems.

[0004] A method for detecting moving objects on water based on unmanned aerial vehicles (UAVs), comprising:

[0005] While the drone is flying along a preset route, the drone acquires visible light images, infrared images, and radar images.

[0006] The visible light image, the infrared image, and the radar image are fused together to obtain a three-dimensional scene model.

[0007] Based on the three-dimensional scene model, the water body boundary contour is extracted, and the water area is determined according to the water body boundary contour;

[0008] Obtain the geometric features and texture patterns of a preset water target, and detect whether the preset water target exists within the water area based on a preset target detection algorithm;

[0009] When the preset water target is present within the water area, the flight of the UAV is controlled according to the motion state of the preset water target.

[0010] In one embodiment, the step of fusing the visible light image, the infrared image, and the radar image to obtain a three-dimensional scene model includes:

[0011] The visible light image, the infrared image, and the radar image are fused using a spatiotemporal registration algorithm to obtain the three-dimensional scene model.

[0012] In one embodiment, the step of extracting the water body boundary contour based on the three-dimensional scene model and determining the water area range based on the water body boundary contour includes:

[0013] The water segmentation module, built on a deep convolutional network, is used to process the image fusion data of the three-dimensional scene model, and the water body boundary contour is extracted using a U-shaped encoder-decoder architecture.

[0014] The extent of the water area is determined based on the water body boundary contour.

[0015] In one embodiment, the step of acquiring the geometric features and texture patterns of a preset water target, and detecting whether the preset water target exists within the water area based on a preset target detection algorithm, includes:

[0016] The geometric features and texture patterns of a preset water target are obtained, and the Faster R-CNN target detection algorithm is used to detect whether there is an object within the water area that matches the geometric features and texture patterns of the preset water target;

[0017] If an object exists within the water area that matches the geometric features and texture pattern of the preset water target, it is identified as the preset water target.

[0018] In one embodiment, the step of acquiring the geometric features and texture patterns of a preset water target, and detecting whether the preset water target exists within the water area based on a preset target detection algorithm, includes:

[0019] The sparse optical flow algorithm is used to predict the motion trajectory of objects within the water area by fusing images of three-dimensional scene models from multiple consecutive frames, thus obtaining the predicted motion trajectory.

[0020] By combining the detection results of the Faster R-CNN target detection algorithm and the detection results of the sparse optical flow algorithm, it is determined whether the preset water target exists within the water area.

[0021] In one embodiment, the step of controlling the flight of the UAV according to the motion state of the preset water target when the preset water target is present includes:

[0022] When the preset water target exists within the water area, the displacement vector of the preset water target is detected using a dense optical flow field calculation method to obtain the motion state of the preset water target, wherein the motion state includes motion speed and motion direction;

[0023] The flight of the UAV is controlled according to the motion state of the preset water target.

[0024] In one embodiment, the step of controlling the flight of the UAV based on the motion state of the preset water target includes:

[0025] When the speed of the preset water target is less than the preset speed, the first preset pathfinding algorithm is invoked to generate a circumferential surveillance route, and the UAV is controlled to fly along the circumferential surveillance route.

[0026] When the change in the speed of the preset water target is greater than the preset acceleration, the second preset pathfinding algorithm is invoked to generate an interception pursuit route, and the UAV is controlled to fly along the interception pursuit route.

[0027] A device for detecting moving objects on water based on a drone, comprising:

[0028] The image acquisition module is used to acquire visible light images, infrared images, and radar images by the UAV when the UAV flies along a preset route.

[0029] An image data fusion module is used to fuse the visible light image, the infrared image, and the radar image to obtain a three-dimensional scene model.

[0030] The water area determination module is used to extract the water body boundary contour based on the three-dimensional scene model, and determine the water area based on the water body boundary contour;

[0031] The target detection module is used to acquire the geometric features and texture patterns of a preset water target, and to detect whether the preset water target exists within the water area based on a preset target detection algorithm.

[0032] The flight control module is used to control the flight of the UAV according to the motion state of the preset water target when the preset water target is present in the water area.

[0033] An electronic device includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to perform the following steps:

[0034] While the drone is flying along a preset route, the drone acquires visible light images, infrared images, and radar images.

[0035] The visible light image, the infrared image, and the radar image are fused together to obtain a three-dimensional scene model.

[0036] Based on the three-dimensional scene model, the water body boundary contour is extracted, and the water area is determined according to the water body boundary contour;

[0037] Obtain the geometric features and texture patterns of a preset water target, and detect whether the preset water target exists within the water area based on a preset target detection algorithm;

[0038] When the preset water target is present within the water area, the flight of the UAV is controlled according to the motion state of the preset water target.

[0039] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0040] While the drone is flying along a preset route, the drone acquires visible light images, infrared images, and radar images.

[0041] The visible light image, the infrared image, and the radar image are fused together to obtain a three-dimensional scene model.

[0042] Based on the three-dimensional scene model, the water body boundary contour is extracted, and the water area is determined according to the water body boundary contour;

[0043] Obtain the geometric features and texture patterns of a preset water target, and detect whether the preset water target exists within the water area based on a preset target detection algorithm;

[0044] When the preset water target is present within the water area, the flight of the UAV is controlled according to the motion state of the preset water target.

[0045] The aforementioned method, device, and electronic equipment for detecting moving objects on water using unmanned aerial vehicles (UAVs) integrate visible light images, infrared images, and radar images to construct a multi-dimensional target discrimination system. This system determines the water area and then detects target vessels within that area. After identifying the target vessels, the UAV is controlled to track them. Compared to existing technologies, this approach better adapts to environmental variables and makes real-time decisions based on target movement for effective tracking, resulting in more accurate detection of water targets. Attached Figure Description

[0046] Figure 1 is a flowchart illustrating a method for detecting moving objects on water based on a drone in one embodiment;

[0047] Figure 2 is a flowchart illustrating a method for detecting moving objects on water based on a drone in another embodiment;

[0048] Figure 3 is an internal structure diagram of an electronic device in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Embodiment 1

[0050] In this embodiment, referring to Figure 1, a method for detecting moving objects on water based on a drone is provided, which includes:

[0051] Step 110: While the UAV is flying along a preset route, the UAV acquires visible light images, infrared images, and radar images.

[0052] In this embodiment, the UAV is equipped with a visible light imager, an infrared thermal imager, and a synthetic aperture radar (SAR). The visible light imager acquires visible light images of the environment, the infrared thermal imager acquires infrared images of the environment, and the SAR acquires radar images of the environment. The radar images are SAR images. It is worth noting that the radar images can also be other types of radar images besides SAR images; this embodiment will not elaborate on these.

[0053] In this embodiment, a preset flight path is used as the basic patrol path, formulated for the target patrol area. It guides the drone's flight in the initial stage of the flight mission. Once the target is detected, the drone's flight path will change according to the detection situation. In addition, during the drone's flight along the preset flight path, a preset speed is also set. Therefore, the drone flies at a preset speed while following the preset flight path.

[0054] Step 120: The visible light image, the infrared image, and the radar image are fused to obtain a three-dimensional scene model.

[0055] In this embodiment, image data from visible light images, infrared images, and synthetic aperture radar images of different bands are fused to obtain a synthetic image of different band data fusion, i.e., a three-dimensional scene model.

[0056] Step 130: Based on the three-dimensional scene model, extract the water body boundary contour, and determine the water area range according to the water body boundary contour.

[0057] It should be understood that in order to accurately identify moving objects on water, it is necessary to accurately identify the water body. In this embodiment, the 3D scene model is analyzed to extract the water body boundary contour, thereby effectively distinguishing the water body from the surrounding terrain features. Based on the water body boundary contour, the water area is determined and marked.

[0058] Step 140: Obtain the geometric features and texture patterns of the preset water target, and detect whether the preset water target exists within the water area based on the preset target detection algorithm.

[0059] It is worth noting that different objects have different geometric features and texture patterns, and different types of ship hulls also have different geometric features and texture patterns. In this embodiment, the geometric features and texture patterns of the preset water target can be collected through target samples, or the geometric features and texture patterns of the target samples can be pre-extracted using machine learning. After obtaining the geometric features and texture patterns of the preset water target, based on the preset target detection algorithm, it is detected whether there is an object within the marked water area that matches the geometric features and texture patterns of the preset water target. If such an object exists, it is identified as the preset water target.

[0060] Step 150: When the preset water target exists within the water area, control the flight of the UAV according to the motion state of the preset water target.

[0061] The motion state of the preset water target includes its direction of motion, speed, and acceleration. In this embodiment, when the preset water target is determined to exist within the water area, it is tracked, and the flight direction, speed, and mode of the UAV are controlled according to the motion state of the preset water target.

[0062] Since the flight speed of the drone is much greater than the speed of the water object, in one embodiment, when the preset water target is present within the water area, the drone is controlled to switch to a low-speed flight mode. The drone's flight is then controlled according to the movement state of the preset water target to track it. In this embodiment, the drone flies at a relatively high speed along a preset route, but switches to a low-speed flight mode upon detecting the preset water target, reducing its speed to match the target's movement speed on the water and maintaining tracking.

[0063] In the above embodiments, a multi-dimensional target discrimination system is constructed by integrating visible light images, infrared images, and radar images to determine the water area. Then, within this water area, target vessels on the water are detected. After identifying the target vessels, the UAV is controlled to fly and track them. Compared to existing technologies, this approach better adapts to environmental variables and can make real-time decisions based on target movement for effective tracking, thus making the detection of water targets more accurate.

[0064] In one embodiment, the step of fusing the visible light image, the infrared image, and the radar image to obtain a three-dimensional scene model includes:

[0065] The visible light image, the infrared image, and the radar image are fused using a spatiotemporal registration algorithm to obtain the three-dimensional scene model.

[0066] In this embodiment, a spatiotemporal registration algorithm is used to spatially align and temporally synchronize the heterogeneous data acquired by visible light, infrared thermal imagers, and synthetic aperture radar. This eliminates the geometric distortions caused by different bands, viewing angles, and sampling time differences in visible light imagers, infrared thermal imagers, and synthetic aperture radar, generating a three-dimensional scene model with a unified coordinate system. This three-dimensional scene model can also be referred to as three-dimensional modal data.

[0067] In one embodiment, the step of extracting the water body boundary contour based on the three-dimensional scene model and determining the water area based on the water body boundary contour includes:

[0068] The water segmentation module, built on a deep convolutional network, is used to process the image fusion data of the three-dimensional scene model, and the water body boundary contour is extracted using a U-shaped encoder-decoder architecture.

[0069] The extent of the water area is determined based on the water body boundary contour.

[0070] In this embodiment, a water segmentation module based on a deep convolutional network is pre-built. Supervised learning is performed using a labeled water dataset. The loss function is a combination of Dice coefficient and cross-entropy loss, and the optimizer is the AdamW algorithm. Data augmentation strategies, including random rotation, brightness adjustment, and noise injection, are introduced during training to improve the model's robustness to changes in lighting and weather interference, thereby obtaining a water segmentation module capable of identifying water bodies and surrounding features.

[0071] In this embodiment, the deep convolutional network adopts a U-shaped encoder-decoder structure, wherein: the encoding layer is composed of multiple convolution-pooling modules stacked together, which gradually extracts hierarchical representations from local edge features to global semantic information; the connection layer fuses shallow high-resolution detail features with deep abstract semantic features across layers; the decoding layer gradually restores spatial resolution through deconvolution operations, and finally outputs pixel-level classification results.

[0072] After inputting the registered 3D modal data: First, multi-scale feature fusion is achieved through Feature Pyramid Network (FPN) to enhance the adaptability to nearshore shallow water areas and open deep water areas; then, attention mechanism is used to automatically focus on the gradient change region at the water-land interface; finally, conditional random field (CRF) post-processing is used to optimize boundary continuity and output an accurate water contour mask.

[0073] In one embodiment, the step of acquiring the geometric features and texture patterns of a preset water target, and detecting whether the preset water target exists within the water area based on a preset target detection algorithm, includes:

[0074] The geometric features and texture patterns of a preset water target are obtained, and the Faster R-CNN target detection algorithm is used to detect whether there is an object within the water area that matches the geometric features and texture patterns of the preset water target;

[0075] If an object exists within the water area that matches the geometric features and texture pattern of the preset water target, it is identified as the preset water target.

[0076] In this embodiment, the geometric features and texture patterns of the pre-defined water targets are obtained by collecting publicly available ship datasets (SeaShips) and using these datasets as samples for training to obtain a pre-trained base model. Subsequently, the detection weights are transferred to the smuggling scenario through domain adaptation technology. A branch for detecting anchor chain hanging objects is added specifically for modified fishing boats, and the correlation modeling between the wake of high-speed boats and the turbulence features under the boat bottom is enhanced.

[0077] In this embodiment, the Feature Pyramid Network (FPN) is introduced into the base network of the Faster R-CNN object detection algorithm to enhance the interaction between low-level detailed features and high-level semantic features through a top-down path. Specifically, in this Faster R-CNN object detection algorithm, the image is first preprocessed. The water area image representing the water area in the fused 3D scene model is preprocessed, including adjusting the image to a preset size and normalizing the image to ensure that the image is suitable for processing by the Faster R-CNN object detection algorithm model. Subsequently, a convolutional neural network is used to extract features from the water area image. In this embodiment, a pre-trained ResNet is used as the backbone network of the convolutional neural network to extract features from the water area image to obtain a high-level feature representation. Then, candidate regions of the water area image are generated using RPN. The candidate regions may contain objects of interest. Through a sliding window and anchor point mechanism, regions with a probability greater than a preset probability of containing a preset water target are selected as target regions from the candidate regions. The target regions are mapped onto a feature map of a preset size, and each target region is classified to determine whether the water object in the target region is a preset water target. At the same time, bounding box regression is performed to accurately locate the preset water target. Finally, non-maximum suppression technology is applied to remove duplicate detection results to ensure that the final output detection result is accurate and unique.

[0078] This improvement effectively solves the problem of insufficient accuracy in detecting small vessels at long distances using traditional Faster R-CNN, enabling the algorithm to simultaneously capture multi-scale features of large near-shore vessels and small speedboats in the open sea. Furthermore, this Faster R-CNN target detection algorithm utilizes historical sample target size data to redesign anchor ratios to three elongated shapes: [1:3, 1:5, 1:7], which are more adaptable to the geometric features of preset water targets than the general model's [1:1, 1:2, 2:1]. In this embodiment, the improved Faster R-CNN algorithm demonstrates excellent accuracy in target detection tasks, maintaining high accuracy even in complex backgrounds. Compared to some more complex detection algorithms, such as Mask R-CNN, Faster R-CNN offers faster processing speed while maintaining accuracy, meeting the needs of real-time monitoring. Moreover, Faster R-CNN can adapt to targets of different scales and shapes, exhibiting good adaptability to the varied shapes and sizes of vessels in aquatic environments.

[0079] In one embodiment, the step of acquiring the geometric features and texture patterns of a preset water target, and detecting whether the preset water target exists within the water area based on a preset target detection algorithm, includes:

[0080] The sparse optical flow algorithm is used to predict the motion trajectory of objects within the water area by fusing images of three-dimensional scene models from multiple consecutive frames, thus obtaining the predicted motion trajectory.

[0081] By combining the detection results of the Faster R-CNN target detection algorithm and the detection results of the sparse optical flow algorithm, it is determined whether the preset water target exists within the water area.

[0082] In this embodiment, within the labeled water area, the Faster R-CNN target detection algorithm is used to achieve multi-scale target localization based on the unique geometric features and texture patterns of the preset water targets. Simultaneously, a sparse optical flow algorithm is combined to predict the motion trajectory of the detection results between consecutive frames, thereby accurately detecting the presence of the preset water targets within the water area. Specifically, Faster R-CNN can accurately identify static targets in an image, while the sparse optical flow algorithm can capture the motion information of the targets. The combination of the two allows for further confirmation of targets based on motion features, building upon static target identification, thus improving detection accuracy.

[0083] In one embodiment, the step of controlling the flight of the UAV based on the motion state of the preset water target when the preset water target is present within the water area includes:

[0084] When the preset water target exists within the water area, the displacement vector of the preset water target is detected using a dense optical flow field calculation method to obtain the motion state of the preset water target, wherein the motion state includes motion speed and motion direction;

[0085] The flight of the UAV is controlled according to the motion state of the preset water target.

[0086] In this embodiment, when it is determined that the preset water target exists within the water area, the displacement vector of the preset water target is detected by the dense optical flow field calculation method, thereby accurately obtaining the movement speed and direction of the preset water target, updating the movement speed and direction of the water target in real time, and then controlling the UAV to track the water target and maintain a better observation angle for the water target.

[0087] It is worth mentioning that in the aforementioned embodiments, the sparse optical flow algorithm is used to detect preset water targets. In this solution, the sparse optical flow algorithm is suitable for scenarios involving target searching and discovery. Therefore, it is necessary to leverage the real-time advantage of sparse optical flow. The sparse optical flow algorithm only performs motion estimation on specific points (such as feature points) in the image, resulting in relatively low computational load and fast processing speed. In scenarios requiring preliminary detection or rapid response, the sparse optical flow algorithm can quickly capture the approximate movement trend of the target.

[0088] Once a preset water target is detected, its displacement vector needs to be accurately estimated. This is where dense optical flow field calculations come in. Dense optical flow field calculations can provide motion information for each pixel in the image, including velocity and direction, thus more accurately describing the object's motion state. The advantages of dense optical flow fields are significant when detailed analysis of an object's trajectory or shape changes is required. Therefore, in this embodiment, using dense optical flow field calculations to detect the displacement vector of the preset water target can accurately obtain the target's motion state, thereby achieving precise tracking of the preset water target.

[0089] In one embodiment, the step of controlling the flight of the UAV based on the motion state of the preset water target includes:

[0090] When the speed of the preset water target is less than the preset speed, the first preset pathfinding algorithm is invoked to generate a circumferential surveillance route, and the UAV is controlled to fly along the circumferential surveillance route.

[0091] When the change in the speed of the preset water target is greater than the preset acceleration, the second preset pathfinding algorithm is invoked to generate an interception pursuit route, and the UAV is controlled to fly along the interception pursuit route.

[0092] In this embodiment, different flight modes are used to track the motion state of a preset water target. Specifically, when the preset water target's speed is less than a preset speed (i.e., the target remains stationary or moves at low speed), the A* pathfinding algorithm is invoked to generate a circling surveillance route, circling the target to ensure the UAV maintains the optimal observation angle. When the preset water target's acceleration is greater than a preset acceleration (i.e., the target is rapidly escaping), the A* pathfinding algorithm is invoked to generate an interception pursuit route, matching the target's acceleration to ensure tracking and accurate image acquisition. In this embodiment, the A* pathfinding algorithm transforms environmental risk factors such as radar detection blind spots and turbulent water areas into a computable cost function, updating the passage risk value of grid nodes in real time through a Bayesian network. When the target vessel accelerates to escape, the search weight of the adjacent 500-meter sea area is automatically increased, forcing the path planning algorithm to prioritize covering high-threat areas. Example 2

[0093] In this embodiment, the vessel is further explained as a preset waterborne target. This embodiment is applied to the detection and tracking of smuggling vessels. Referring to Figure 2, the implementation process of this UAV-based method for detecting moving objects on water is as follows:

[0094] The first step is to establish a basic drone patrol route plan based on the basic direction of the waterway and the key monitoring locations of suspicious vessels. During the execution of the drone patrol route, the drone first performs routine inspection tasks along the predetermined basic route. During this process, the three core payloads—visible light imager, infrared thermal imager, and synthetic aperture radar (SAR)—are simultaneously activated for collaborative observation. Through spatiotemporal registration algorithms, image data acquired in different bands are fused into a unified three-dimensional scene model.

[0095] The second step involves a water segmentation module built on a deep convolutional network to process the fused data. Then, a U-shaped encoder-decoder architecture is used to accurately extract the water body boundary contours, effectively distinguishing natural water surfaces from surrounding land features.

[0096] 1. Existing registration algorithms are used to spatially align and temporally synchronize heterogeneous data acquired by visible light, infrared thermal imagers, and synthetic aperture radar (SAR), eliminating geometric distortions caused by differences in viewing angles and sampling time differences between sensors of different bands, and generating a three-dimensional scene model with a unified coordinate system.

[0097] 2. The network adopts a U-shaped encoder-decoder structure, wherein:

[0098] The encoding layer consists of multiple stacked convolutional-pooling modules, which gradually extract hierarchical representations from local edge features to global semantic information; the connection layer fuses shallow high-resolution detail features with deep abstract semantic features across layers; the decoding layer gradually restores spatial resolution through deconvolution operations, and finally outputs pixel-level classification results.

[0099] 3. Supervised learning was performed using a labeled water dataset. The loss function employed was a combination of the Dice coefficient and cross-entropy loss, and the AdamW algorithm was selected as the optimizer. Data augmentation strategies, including random rotation, brightness adjustment, and noise injection, were introduced during training to improve the model's robustness to changes in illumination and weather disturbances.

[0100] 4. After inputting the registered trimodal data: First, multi-scale feature fusion is achieved through Feature Pyramid Network (FPN) to enhance the adaptability to nearshore shallow water areas and open deep water areas; then, attention mechanism is used to automatically focus on the gradient change region at the water-land interface; finally, the boundary continuity is optimized by Conditional Random Field (CRF) post-processing to output an accurate water contour mask.

[0101] The third step is to deploy the Faster R-CNN target detection algorithm within the marked water area, combine the unique geometric features and texture patterns of ships to achieve multi-scale target localization, and introduce a sparse optical flow algorithm to predict the motion trajectory of the detection results between consecutive frames.

[0102] Faster R-CNN can accurately identify static targets in images, while the sparse optical flow algorithm can capture the motion information of the targets. Combining the two allows for further target confirmation through motion features, building upon static recognition, thereby improving detection accuracy.

[0103] In this Faster R-CNN object detection algorithm, a Feature Pyramid Network (FPN) is introduced into the base network to enhance the interaction between low-level detailed features and high-level semantic features through a top-down path. This improvement effectively solves the problem of insufficient detection accuracy of traditional Faster R-CNN for small vessels at long distances, enabling the algorithm to simultaneously capture multi-scale features of large near-shore vessels and small speedboats in the open sea. Based on the aspect ratio distribution statistics of vessels in historical smuggling case datasets, the anchor ratios are redesigned to three narrow shapes: [1:3, 1:5, 1:7], which are more suitable for the geometric features of typical smuggling vehicles such as high-speed yachts and modified fishing boats than the general model's [1:1, 1:2, 2:1].

[0104] Specifically, the Faster R-CNN object detection algorithm's detection process is as follows: 1. First, image preprocessing: The input drone-captured water images are preprocessed, including resizing and normalization, to ensure the images are suitable for model processing. 2. Then, a convolutional neural network is used to extract features from the image. This method uses a pre-trained ResNet as the backbone network to obtain high-level feature representations. 3. Next, RPN is used to generate candidate regions, which may contain objects of interest. A sliding window and anchor point mechanism are used to quickly filter out regions with a high probability of containing the target. 4. The candidate regions are mapped onto a fixed-size feature map, and then each candidate region is classified to determine whether it is a smuggling vessel. Bounding box regression is also performed to accurately locate the vessel's position. 5. Finally, non-maximum suppression is applied to remove duplicate detection results, ensuring that the final output detection result is accurate and unique.

[0105] The Faster R-CNN object detection algorithm has the following advantages:

[0106] 1. Faster R-CNN performs well in object detection tasks, especially in complex backgrounds where it maintains high accuracy, which is particularly important for identifying smuggling vessels in waterways.

[0107] 2. Compared to some more complex detection algorithms, such as Mask R-CNN, Faster R-CNN maintains accuracy while having a faster processing speed, which can meet the needs of real-time monitoring.

[0108] 3. Faster R-CNN can adapt to targets of different scales and shapes, and has good adaptability to the varied shapes and sizes of ships in aquatic environments.

[0109] The geometric features and texture patterns of smuggling vessels can be obtained by collecting publicly available ship datasets (SeaShips) and using these datasets as samples for training to obtain a pre-trained base model. Subsequently, the detection weights are transferred to the smuggling scenario through domain adaptation techniques. A branch for detecting anchor chain hanging objects is added specifically for modified fishing boats, and the correlation model between the wake of high-speed boats and the turbulence features under the boat bottom is enhanced.

[0110] Fourth, when the system determines that there is a valid target, it automatically switches to low-speed flight mode, uses dense optical flow field calculation method to quantify and analyze the target displacement vector, and updates its velocity parameters and direction of motion in real time; if the target remains stationary or moves at low speed, the A* pathfinding algorithm is called to generate a circling surveillance route to ensure that the UAV always maintains the best observation angle; for target carriers that suddenly accelerate away, the A* pathfinding algorithm is called to generate an interception pursuit route.

[0111] In the A* pathfinding algorithm, environmental risk factors such as radar detection blind spots and areas with strong currents are transformed into a computable cost function, and the passage risk value of grid nodes is updated in real time through a Bayesian network. When the target vessel accelerates to escape, the system automatically increases the search weight of the adjacent 500-meter sea area, forcing the path planning algorithm to prioritize covering high-threat areas.

[0112] It's worth noting that in target detection scenarios, the real-time advantage of sparse optical flow can be leveraged. Sparse optical flow algorithms only estimate motion at specific points (such as feature points) in the image, resulting in relatively low computational cost and fast processing speed. In initial detection or scenarios requiring rapid response, sparse optical flow algorithms can quickly capture the approximate motion trend of the target.

[0113] Once a target is detected, its displacement vector needs to be accurately estimated. This is where dense optical flow fields come in. Dense optical flow field calculation methods can provide motion information for each pixel in the image, including velocity and direction, thus more accurately describing the object's motion state. The advantages of dense optical flow fields are evident when detailed analysis of an object's trajectory or shape changes is required.

[0114] This invention achieves fully automated processing from data acquisition to action decision-making by constructing a closed-loop feedback control system. It breaks through the performance bottleneck of single-mode by relying on multi-sensor fusion technology and improves the adaptability in complex scenarios by using an advanced algorithm framework, ultimately forming an intelligent inspection system that can meet the needs of different working conditions.

[0115] It should be understood that although the steps in the flowchart of Figure 1 are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in Figure 1 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps. Example 3

[0116] In this embodiment, a device for detecting moving objects on water based on a drone is provided, comprising:

[0117] The image acquisition module is used to acquire visible light images, infrared images, and radar images by the UAV when the UAV flies along a preset route.

[0118] An image data fusion module is used to fuse the visible light image, the infrared image, and the radar image to obtain a three-dimensional scene model.

[0119] The water area determination module is used to extract the water body boundary contour based on the three-dimensional scene model, and determine the water area based on the water body boundary contour;

[0120] The target detection module is used to acquire the geometric features and texture patterns of a preset water target, and to detect whether the preset water target exists within the water area based on a preset target detection algorithm.

[0121] The flight control module is used to control the flight of the UAV according to the motion state of the preset water target when the preset water target is present in the water area.

[0122] In one embodiment, the image data fusion module is further configured to fuse the visible light image, the infrared image, and the radar image using a spatiotemporal registration algorithm to obtain the three-dimensional scene model.

[0123] In one embodiment, the water area determination module includes:

[0124] The water body contour extraction unit is used to process the image fusion data of the three-dimensional scene model using a water segmentation module built based on a deep convolutional network, and to extract the water body boundary contour using a U-shaped encoder-decoder architecture.

[0125] A water area determination unit is used to determine the water area based on the water body boundary contour.

[0126] In one embodiment, the target detection module includes:

[0127] The target feature detection unit is used to acquire the geometric features and texture patterns of a preset water target, and based on the Faster R-CNN target detection algorithm, detect whether there is an object within the water area that matches the geometric features and texture patterns of the preset water target;

[0128] The target determination unit is used to determine an object as the preset water target if there is an object within the water area whose geometric features and texture patterns match the preset water target.

[0129] In one embodiment, the target detection module includes:

[0130] The trajectory prediction calculation unit is used to predict the motion trajectory of objects within the water area by using a sparse optical flow algorithm to fused images of three-dimensional scene models from multiple consecutive frames, and to obtain the predicted motion trajectory.

[0131] The target determination unit is also used to combine the detection results of the Faster R-CNN target detection algorithm and the detection results of the sparse optical flow algorithm to determine whether the preset water target exists within the water area.

[0132] In one embodiment, the flight control module includes:

[0133] The motion state calculation unit is used to detect the displacement vector of the preset water target using a dense optical flow field calculation method when the preset water target exists within the water area, and to obtain the motion state of the preset water target, wherein the motion state includes motion speed and motion direction;

[0134] The flight control unit is used to control the flight of the UAV based on the motion state of the preset water target.

[0135] In one embodiment, the root flight control unit includes:

[0136] The orbital flight control subunit is used to call the first preset pathfinding algorithm to generate an orbital surveillance route when the speed of the preset water target is less than the preset speed, and control the UAV to fly along the orbital surveillance route.

[0137] The pursuit flight control subunit is used to call the second preset pathfinding algorithm to generate an interception pursuit route when the change in the speed of the preset water target is greater than the preset acceleration, and control the UAV to fly along the interception pursuit route.

[0138] Specific limitations regarding the UAV-based detection device for moving objects on water can be found in the above-described limitations of the UAV-based detection method for moving objects on water, and will not be repeated here. Each unit in the aforementioned UAV-based detection device for moving objects on water can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in the electronic device, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each unit. Example 4

[0139] In this embodiment, an electronic device is provided. Its internal structure is shown in Figure 3. The electronic device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs, and a database is deployed on the non-volatile storage medium. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the electronic device is used to communicate with other electronic devices that have deployed application software. When the computer program is executed by the processor, it implements a method for detecting moving objects on water based on a drone. The display screen of the electronic device can be a liquid crystal display (LCD) or an e-ink display. The input device of the electronic device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the outer casing of the electronic device, or an external keyboard, touchpad, or mouse, etc.

[0140] Those skilled in the art will understand that the structure shown in Figure 3 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0141] In one embodiment, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the UAV-based method for detecting moving objects on water in any of the above embodiments. Embodiment Five

[0142] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps in the method for detecting moving objects on water based on unmanned aerial vehicles in any of the above embodiments.

[0143] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for detecting moving objects on water based on unmanned aerial vehicles (UAVs), characterized in that, This includes: acquiring visible light images, infrared images, and radar images using the drone while it is flying along a preset route; The visible light image, the infrared image, and the radar image are fused to obtain a three-dimensional scene model. Based on the three-dimensional scene model, the water body boundary contour is extracted, and the water area is determined according to the water body boundary contour. The geometric features and texture patterns of a preset water target are obtained, and the presence of the preset water target within the water area is detected based on a preset target detection algorithm. When the preset water target is present within the water area, the flight of the UAV is controlled according to the motion state of the preset water target. The step of acquiring the geometric features and texture patterns of a preset water target, and detecting whether the preset water target exists within the water area based on a preset target detection algorithm, includes: using a sparse optical flow algorithm to predict the motion trajectory of objects within the water area from a fused image of a multi-frame 3D scene model, obtaining a predicted motion trajectory; combining the detection results of the Faster R-CNN target detection algorithm and the detection results of the sparse optical flow algorithm to determine whether the preset water target exists within the water area; the step of controlling the flight of the UAV according to the motion state of the preset water target when the preset water target exists within the water area includes: when the preset water target exists within the water area, using a dense optical flow field calculation method to detect the displacement vector of the preset water target, obtaining the motion state of the preset water target, wherein the motion state includes motion speed and motion direction; and controlling the flight of the UAV according to the motion state of the preset water target.

2. The method according to claim 1, characterized in that, The step of fusing the visible light image, the infrared image, and the radar image to obtain a three-dimensional scene model includes: fusing the visible light image, the infrared image, and the radar image using a spatiotemporal registration algorithm to obtain the three-dimensional scene model.

3. The method according to claim 1, characterized in that, The steps of extracting water body boundary contours based on the three-dimensional scene model and determining the water area range based on the water body boundary contours include: processing the image fusion data of the three-dimensional scene model using a water area segmentation module built based on a deep convolutional network, extracting the water body boundary contours using a U-shaped encoder-decoder architecture, and determining the water area range based on the water body boundary contours.

4. The method according to claim 1, characterized in that, The step of obtaining the geometric features and texture patterns of a preset water target, and detecting whether the preset water target exists within the water area based on a preset target detection algorithm, includes: obtaining the geometric features and texture patterns of the preset water target, and detecting whether there is an object within the water area that matches the geometric features and texture patterns of the preset water target based on the Faster R-CNN target detection algorithm; when there is an object within the water area that matches the geometric features and texture patterns of the preset water target, it is determined to be the preset water target.

5. The method according to claim 1, characterized in that, The step of controlling the flight of the UAV according to the motion state of the preset water target includes: when the motion speed of the preset water target is less than the preset speed, a first preset pathfinding algorithm is invoked to generate a circling surveillance route, and the UAV is controlled to fly along the circling surveillance route; when the change in the motion speed of the preset water target is greater than the preset acceleration, a second preset pathfinding algorithm is invoked to generate an interception pursuit route, and the UAV is controlled to fly along the interception pursuit route.

6. A device for detecting moving objects on water based on unmanned aerial vehicles (UAVs), characterized in that, include: The image acquisition module is used to acquire visible light images, infrared images, and radar images by the UAV when the UAV flies along a preset route. The image data fusion module is used to fuse the visible light image, the infrared image, and the radar image to obtain a three-dimensional scene model; the water area range determination module is used to extract the water body boundary contour based on the three-dimensional scene model and determine the water area range based on the water body boundary contour; the target detection module is used to acquire the geometric features and texture patterns of a preset water target and detect whether the preset water target exists within the water area based on a preset target detection algorithm. The flight control module is used to control the flight of the UAV according to the motion state of the preset water target when the preset water target is present in the water area; The target detection module includes: a trajectory prediction calculation unit, used to predict the motion trajectory of objects within the water area using a sparse optical flow algorithm on a fused image of a multi-frame 3D scene model, to obtain a predicted motion trajectory; a target determination unit is also used to combine the detection results of the Faster R-CNN target detection algorithm and the detection results of the sparse optical flow algorithm to determine whether the preset water target exists within the water area; the flight control module includes: a motion state calculation unit, used to detect the displacement vector of the preset water target using a dense optical flow field calculation method when the preset water target exists within the water area, to obtain the motion state of the preset water target, wherein the motion state includes motion speed and motion direction; and a flight control unit, used to control the flight of the UAV according to the motion state of the preset water target.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images