Interactive water surface drowning person identification and positioning method and system

Abstract

The invention provides an interactive recognition and positioning method and system for people falling into water on the water surface. The method comprises the steps that a video acquisition module is started, and people falling into water on the water surface are searched for through a people-falling-into-water recognition module; a drowning person target positioning module is used to calculate camera coordinates of drowning persons, and a high-precision RTK positioning module is combined to convert geodetic coordinates of drowning person targets and send the geodetic coordinates to the unmanned ship; the unmanned ship decelerates after approaching a preset range of a drowning person target, a video acquisition module at a horizontal visual angle is started to acquire information of the drowning person, the drowning person target identification module is used for identifying the drowning person, and a thermal imaging camera and laser radar ranging information are fused to further identify and position a water surface target; and fusing the positioning information of the drowning personnel provided by the unmanned aerial vehicle and the unmanned ship by using a Kalman filter to obtain the accurate geodetic coordinates of the drowning personnel. The method has good practicability.

Description

Technical Field

[0001] The present invention relates to the field of positioning and identification technology, and specifically, to an interactive method and system for identifying and locating a person who has fallen into water; more specifically, to a method and system for jointly identifying and locating a person who has fallen into water by interactively utilizing an unmanned boat-aircraft cross-domain platform. Background Art

[0002] With the frequent occurrence of accidents on the water, the recognition technology for people who fall into the water in complex and dangerous waters is increasingly needed, and the recognition technology for people who fall into the water is also developing towards intelligent and unmanned operation. Traditional recognition of people who fall into the water has problems such as low recognition efficiency and high human resource consumption. The recognition of people who fall into the water in complex and dangerous waters will endanger the lives of workers. Therefore, the efficiency of identifying people who fall into the water can be improved by establishing a joint recognition system of air and sea heterogeneous unmanned systems, forming air and sea stereoscopic observation technology, and reducing the cost of identifying people who fall into the water.

[0003] In the air and sea heterogeneous unmanned system identification system, unmanned boats and drones work closely together and complement each other's strengths. Drones give full play to their advantages of long communication distance and wide viewing angle, and play the main search role in the tasks of identifying and positioning targets in water areas. They are equipped with self-stabilizing platform cameras to identify and locate people who fall into the water at the scene of water accidents. At the same time, unmanned boats need to have the capabilities of traditional manned search ships and can provide energy supply stations and communication stations for drones.

[0004] In complex scenes, the target image of people falling into the water is affected by factors such as wind and waves and changes in lighting. The target has weak appearance features, small size, and diverse shapes, making it difficult to effectively detect the target using traditional machine vision methods. With the development of computer vision and information fusion technology, the use of deep learning-based recognition and positioning technology for people falling into the water can greatly improve the search efficiency and avoid the problem of missing targets caused by fatigue and carelessness of search personnel during traditional manual search for people falling into the water. Summary of the invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide an interactive method and system for identifying and locating people falling into the water.

[0006] An interactive method for identifying and locating a person who has fallen into water provided by the present invention comprises:

[0007] Step S1: Start the video acquisition module and search for the person who fell into the water through the person who fell into the water recognition module;

[0008] Step S2: using the man-in-water target positioning module to calculate the camera coordinates of the man in the water, and combining with the high-precision RTK positioning module to convert the geodetic coordinates of the man in the water target and send them to the unmanned boat;

[0009] Step S3: The unmanned boat slows down after approaching the preset range of the person falling into the water, and starts the video acquisition module of the horizontal perspective to obtain the information of the person falling into the water, uses the person falling into the water target recognition module to identify the person falling into the water, and integrates the thermal imaging camera and laser radar ranging information to further identify and locate the surface target;

[0010] Step S4: Use the Kalman filter to fuse the positioning information of the person in the water provided by the UAV and the unmanned boat to obtain the precise geodetic coordinates of the person in the water.

[0011] Preferably, in step S1:

[0012] The implementation steps of the drowning person identification module are as follows:

[0013] Establish data sets through three methods: existing public data sets, virtual simulation data sets, and self-made data sets;

[0014] The public data set is collected by consulting literature to obtain a data set for identifying people falling into the water, including the SeaDroneSee data set;

[0015] The virtual simulation data set provides a basis for the construction of parallel surface drowning person identification targets between the virtual world and the real world through the virtual simulation platform;

[0016] The self-made dataset collects the target of people falling into the water on the spot through the video acquisition module equipped with drones and unmanned boats. Considering the richness of the dataset in different environments, the floating state and swimming posture of the person on the water are used as the falling posture and used as the image source of the dataset, and the self-made video images are preprocessed. The video acquisition module includes visible light images, thermal imaging images, and infrared images.

[0017] Preferably, the video image is preprocessed, the video containing the target of falling into the water is frame-processed, each frame of the image containing the target of falling into the water is obtained, the noise is removed by using a median filter, the contrast is enhanced by using histogram equalization, and each frame of the image containing the target of falling into the water is labeled by using LabelImg software, and the images in which the degree of obviousness of the target of falling into the water is lower than the preset standard or the degree of image blurring is higher than the preset standard, which affect the training results, are deleted;

[0018] Based on the improved YOLOv8 model pre-training, the pre-processed image dataset is resized to 640*640 as the YOLOv8 model input; by running the YOLOv8 model, the training results of each image dataset are obtained, including confidence, frame rate, and category information; by improving the YOLOv8 head network, setting the confidence threshold, and filtering the image data with low confidence, the training recognition results are obtained;

[0019] Model migration and deployment: deploy the trained model weight file to the edge computing devices carried by drones and unmanned boats, and use the video acquisition modules carried by each to autonomously identify the target video images of people falling into the water captured in real time through the pre-trained YOLOv8 model, and further optimize the model parameters based on the recognition results.

[0020] Preferably, in step S2:

[0021] The implementation steps of the drowning person target positioning module are as follows:

[0022] The video acquisition module is calibrated to obtain the equivalent focal length of the horizontal and vertical axes of the video acquisition module and the coordinates of the main point of the window; the drone uses RTK to obtain the altitude and longitude and latitude above the ground during the flight, and the target detection algorithm is used to identify the person falling into the water and obtain the position information of the target falling into the water in the image;

[0023] The distance between the target point and the image principal point in the horizontal axis and the vertical axis direction in the imaging plane is obtained according to the equivalent focal length of the video acquisition module and the pixel information from the target point to the image principal point;

[0024] The pixel distance between the target point and the main point of the window in the horizontal and vertical directions is calculated using the principle of similar triangles, and the coordinates of the target point in the coordinate system of the video acquisition module are obtained;

[0025] The coordinates of the target point in the video acquisition module coordinate system are converted into the coordinates in the earth coordinate system.

[0026] Preferably, the unmanned boat fuses the laser radar data and the video acquisition module data through the Kalman filter algorithm to obtain the relative position information of the target for identifying the person in the water, and then converts it through the positioning data of the boat body itself to obtain the geodetic coordinates of the target for identifying the person in the water;

[0027] The target location data located by the UAV and the unmanned boat are fused using the triangulation positioning algorithm to obtain the positioning data of the target falling into the water;

[0028] Dynamically adjust the search strategy: The UAV dynamically adjusts the flight altitude according to the confidence of the identified target, and the unmanned boat maintains the preset escort speed and tracking distance according to the flight altitude and cruising speed of the UAV to ensure communication quality.

[0029] According to an interactive water surface drowning person identification and positioning system provided by the present invention, the interactive water surface drowning person identification and positioning method is implemented, including:

[0030] Video acquisition module: The acquisition module obtains the information of the person who falls into the water, and cooperates with the person who falls into the water identification module to search for the person who falls into the water;

[0031] Falling-in-water target positioning module: Calculate the camera coordinates of the falling-in-water person

[0032] High-precision RTK positioning module: Combined with the man-in-water positioning module, the geodetic coordinates of the man-in-water target are calculated and sent to the unmanned boat;

[0033] Drowned Person Target Identification Module: Identify drowned persons and integrate thermal imaging camera and lidar ranging information to further identify and locate surface targets.

[0034] Preferably, in the module for identifying people falling into water:

[0035] Establish data sets through three methods: existing public data sets, virtual simulation data sets, and self-made data sets;

[0036] The public data set is collected by consulting literature to obtain a data set for identifying people falling into the water, including the SeaDroneSee data set;

[0037] The virtual simulation data set provides a basis for the construction of parallel surface drowning person identification targets between the virtual world and the real world through the virtual simulation platform;

[0038] The self-made dataset collects the target of people falling into the water on the spot through the video acquisition module equipped with drones and unmanned boats. Considering the richness of the dataset in different environments, the floating state and swimming posture of the person on the water are used as the falling posture and used as the image source of the dataset, and the self-made video images are preprocessed. The video acquisition module includes visible light images, thermal imaging images, and infrared images.

[0039] Preferably, the video image is preprocessed, the video containing the target of falling into the water is frame-processed, each frame of the image containing the target of falling into the water is obtained, the noise is removed by using a median filter, the contrast is enhanced by using histogram equalization, and each frame of the image containing the target of falling into the water is labeled by using LabelImg software, and the images in which the degree of obviousness of the target of falling into the water is lower than the preset standard or the degree of image blurring is higher than the preset standard, which affect the training results, are deleted;

[0040] Based on the improved YOLOv8 model pre-training, the pre-processed image dataset is resized to 640*640 as the YOLOv8 model input; by running the YOLOv8 model, the training results of each image dataset are obtained, including confidence, frame rate, and category information; by improving the YOLOv8 head network, setting the confidence threshold, and filtering the image data with low confidence, the training recognition results are obtained;

[0041] Model migration and deployment: deploy the trained model weight file to the edge computing devices carried by drones and unmanned boats, and use the video acquisition modules carried by each to autonomously identify the target video images of people falling into the water captured in real time through the pre-trained YOLOv8 model, and further optimize the model parameters based on the recognition results.

[0042] Preferably, in the drowning person target positioning module:

[0043] The video acquisition module is calibrated to obtain the equivalent focal length of the horizontal and vertical axes of the video acquisition module and the coordinates of the main point of the window; the drone uses RTK to obtain the altitude and longitude and latitude above the ground during the flight, and the target detection algorithm is used to identify the person falling into the water and obtain the position information of the target falling into the water in the image;

[0044] The distance between the target point and the image principal point in the horizontal axis and the vertical axis direction in the imaging plane is obtained according to the equivalent focal length of the video acquisition module and the pixel information from the target point to the image principal point;

[0045] The pixel distance between the target point and the main point of the window in the horizontal and vertical directions is calculated using the principle of similar triangles, and the coordinates of the target point in the coordinate system of the video acquisition module are obtained;

[0046] The coordinates of the target point in the video acquisition module coordinate system are converted into the coordinates in the earth coordinate system.

[0047] Preferably, the unmanned boat fuses the laser radar data and the video acquisition module data through the Kalman filter algorithm to obtain the relative position information of the target for identifying the person in the water, and then converts it through the positioning data of the boat body itself to obtain the geodetic coordinates of the target for identifying the person in the water;

[0048] The target location data located by the UAV and the unmanned boat are fused using the triangulation positioning algorithm to obtain the positioning data of the target falling into the water;

[0049] Dynamically adjust the search strategy: The UAV dynamically adjusts the flight altitude according to the confidence of the identified target, and the unmanned boat maintains the preset escort speed and tracking distance according to the flight altitude and cruising speed of the UAV to ensure communication quality.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. The present invention provides an autonomous recognition and positioning method for people falling into water based on improved YOLOv8, which can be applied to complex and dangerous areas for searching people falling into water. The application area of ​​the target recognition module for people falling into water carried by drones includes but is not limited to water areas, mountainous areas and other environments that are difficult for humans to reach quickly.

[0052] 2. The present invention provides an interactive method for identifying and locating people who have fallen into the water based on a heterogeneous unmanned system. The method is based on the information interaction between unmanned boats and unmanned aerial vehicles, and adaptively generates strategies for identifying and locating people who have fallen into the water in different scenarios, which helps to improve the efficiency of searching for people who have fallen into the water and enhance the collaborative operation capability of unmanned platforms.

[0053] 3. The present invention utilizes the image information of the person who falls into the water and integrates the navigation and positioning information of heterogeneous unmanned systems to calculate accurate target positioning information, which has good practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0055] Figure 1 It is a schematic diagram of the surface target recognition process of the heterogeneous unmanned system of the present invention;

[0056] Figure 2 A schematic diagram of surface target recognition training results of a heterogeneous unmanned system of the present invention;

[0057] Figure 3 A schematic diagram of various coordinate systems for positioning surface targets of a heterogeneous unmanned system of the present invention;

[0058] Figure 4 A schematic diagram of the coordinate system of the surface target positioning video acquisition module of the heterogeneous unmanned system of the present invention;

[0059] Figure 5 It is a schematic diagram of pinhole imaging for positioning surface targets of the heterogeneous unmanned system of the present invention;

[0060] Figure 6 It is a flow chart of surface target positioning of the heterogeneous unmanned system of the present invention;

[0061] Figure 7 This is a flow chart of the surface target positioning camera and lidar data fusion detection of the heterogeneous unmanned system of the present invention. DETAILED DESCRIPTION

[0062] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0063] Embodiment 1:

[0064] The purpose of the present invention is to provide an interactive method for identifying and locating people who have fallen into the water, in order to address the problem of missing targets due to fatigue and carelessness of search personnel in the traditional manual search for people who have fallen into the water, so as to achieve rapid identification and positioning of people who have fallen into the water.

[0065] like Figure 1-Figure 7 The technical implementation approach adopted by the purpose of the present invention is: an interactive method for identifying and locating people falling into the water, which is characterized by using heterogeneous unmanned platforms to interactively identify and locate surface targets, and the technical solution adopted is as follows:

[0066] The heterogeneous unmanned system described in the present invention consists of three parts: an unmanned aerial vehicle, an unmanned boat, and a shore-based monitoring station. The unmanned boat carries the unmanned aerial vehicle to the waters to be patrolled. After the shore-based monitoring station issues a search command, the unmanned aerial vehicle is lifted to a specified height, and the video acquisition module is turned on and the person-in-water recognition module is used to search for the person who falls into the water. The camera coordinates of the person who falls into the water are solved using the person-in-water target positioning module, and the geodetic coordinates of the person-in-water target are converted in combination with the high-precision RTK positioning module and sent to the unmanned boat. The above process is completed by an edge computing device; the unmanned boat slows down after approaching the person-in-water target within a certain range, and turns on the video acquisition module with a horizontal perspective to obtain the person-in-water information, and uses the person-in-water target recognition module to identify the person who falls into the water, and fuses the thermal imaging camera and lidar ranging information to further identify and locate the surface target, and finally uses the Kalman filter to fuse the person-in-water positioning information provided by the unmanned aerial vehicle and the unmanned boat to obtain the precise geodetic coordinates of the person who falls into the water. The above process is completed by an edge computing device.

[0067] According to the above scheme, the implementation steps of the drowning person identification module are as follows:

[0068] Step 101: Establish a data set through three methods: existing public data sets, virtual simulation data sets, and self-made data sets;

[0069] The collection of public data sets, such as the SeaDroneSee data set (Varga et al., 2022), aims to promote the use of drones for search and identification of people in the sea. The data set has systematically annotated more than 54,000 frames of images, including about 400,000 target instances at different heights of 5 to 260m and different viewing angles of 0 to 90°. In addition, the data set also provides additional image annotations such as shooting height and viewing angle;

[0070] The virtual simulation data set provides the possibility for parallel construction of water surface drowning person identification targets between the virtual world and the real world through the virtual simulation platform. The virtual simulation platform constructs the data set to solve problems such as data imbalance, insufficient scene diversity, and small samples;

[0071] The self-made data set is collected on-site by drones and unmanned boats equipped with video acquisition modules to identify people falling into the water. Considering the richness of data sets in different environments, the floating state and swimming posture of people on the water are approximately regarded as the falling posture and used as the image source of the data set, and the self-made video images are preprocessed. The video acquisition module includes visible light images, thermal imaging images, and infrared images;

[0072] Step 102: Preprocessing the video image, performing frame processing on the video containing the target of drowning person identification, obtaining each frame of the image containing the target of drowning person identification, removing noise using a median filter, using histogram equalization to enhance contrast, and then using LabelImg software to label each frame of the image containing the drowning person, deleting the images in which the target of drowning person identification is not obvious or the image is blurred, etc., which affect the training results;

[0073] Step 103: Based on the improved YOLOv8 model pre-training, firstly, the image dataset pre-processed in the above step 2 is resized to adjust the image size to 640*640 as the YOLOv8 model input; secondly, the training results of each image dataset are obtained by running the YOLOv8 model, including confidence, frame rate, and category information; finally, the YOLOv8 head network is improved, the confidence threshold is set, and the image data with low confidence is filtered to obtain the final training recognition result;

[0074] Step 104: Model migration and deployment. After completing step 3 above, the trained model weight file is deployed to the edge computing devices carried by the drone and the unmanned boat. The real-time video images of the target video images of the drowning person are autonomously identified through the pre-trained YOLOv8 model through the video acquisition modules carried by each device. The model parameters are further optimized according to the identification results.

[0075] Further, according to the above scheme, the implementation steps of the drowning person target positioning module are as follows:

[0076] Step 201: First, the video acquisition module is calibrated to obtain the equivalent focal lengths of the horizontal and vertical axes of the video acquisition module and the coordinates of the main point of the window; the drone uses RTK to obtain the altitude and longitude and latitude above the ground during the flight, and starts the target detection algorithm to identify the person falling into the water, and obtain the position information of the target falling into the water in the image;

[0077] Step 202: The distance between the target point and the image principal point in the horizontal and vertical directions in the imaging plane can be obtained according to the equivalent focal length of the video acquisition module and the pixel information from the target point to the image principal point;

[0078] Step 203: using the principle of similar triangles to obtain the pixel distance between the target point and the main point of the window in the horizontal and vertical axis directions, the coordinates of the target point in the video acquisition module coordinate system can be obtained;

[0079] Step 204: coordinates of the target point in the video acquisition module coordinate system are converted to obtain coordinates in the earth coordinate system;

[0080] Step 205: Further, the unmanned boat fuses the laser radar data and the video acquisition module data through the Kalman filter algorithm to obtain the relative position information of the target for identifying the person in the water, and then converts it through the positioning data of the boat body itself to obtain the geodetic coordinates of the target for identifying the person in the water;

[0081] Step 206: The target position data located by the UAV and the unmanned boat are integrated using a triangulation positioning algorithm to obtain the final positioning data of the target falling into the water;

[0082] Step 207: Dynamically adjust the search strategy. The present invention proposes a method for dynamically adjusting the flight altitude and cruising speed of the UAV according to factors such as the visibility of people falling into the water and the communication quality. Furthermore, the UAV dynamically adjusts the flight altitude according to the confidence of the identified target, and the unmanned boat maintains a certain escort speed and tracking distance according to the flight altitude and cruising speed of the UAV to ensure the communication quality.

[0083] Embodiment 2:

[0084] Embodiment 2 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0085] This embodiment provides an interactive surface target recognition and positioning method. The process of identifying a person who falls into the water is as follows: Figure 1 As shown, further, the specific steps of identifying the target of a person falling into the water are as follows:

[0086] Step 101: Establish a data set. By collecting existing public data sets, virtual simulation data sets, and self-made data sets, a total of 60,000 data sets are obtained, including 4 types of objects: people, lifebuoys, life jackets, and buoys. The data sets include water surface target data sets during the day and at night to ensure that the target recognition module can meet the recognition capabilities under all-weather conditions;

[0087] Step 102: Labeling the image of the person falling into the water. The LabelImg software is used to label the image. The LabelImg software is a commonly used image labeling software based on the Python language. In the LabelImg software interface, the person falling into the water in the data set is marked as "Floater" by manually selecting a rectangular box to complete the labeling.

[0088] Step 103: Building a model pre-training environment. The target recognition model is built based on the PyTorch framework. All model training and testing work is completed under the Windows system. The CPU model of the model pre-training device is Intel (R) i9-12900, and the GPU model is NVIDIA RTX A4500. Considering that the actual application scenario is that the drone platform is equipped with an edge computer, a smaller algorithm model has lower requirements on the GPU of the mobile device and is more friendly. The present invention adopts an improved YOLOv8-s model with smaller model parameters;

[0089] Boxes = YOLOv8 (I enhanced ) (1)

[0090] Among them, I enhanced is the enhanced image, boxes is a list of detection boxes, each detection box contains the position (x, y, w, h) and confidence c;

[0091] Step 104: Data preprocessing. In order to improve the recognition accuracy of the YOLOv8-s model, the preprocessing of the data set is crucial. The data set preprocessing mainly includes image rotation, flipping, cropping and other enhancements as well as filtering and noise reduction. Image preprocessing is to perform median filtering and denoising on the image output by the video acquisition module. The specific expression is as follows:

[0092] I filtered (x,y)=median(I(xk,y (2)

[0093] -k))

[0094] Among them, I is the input image, k is the radius of the filter, and median is the median function;

[0095] I enhanced (x,y)=H(I(x,y)) (3)

[0096] Where I is the input image and H is the cumulative distribution function;

[0097] Step 105: Model pre-training results. During the model training process, the ratio of training set: validation set: test set is set to 7:2:1, the input image is 640×640, the round is 100, the early stopping mechanism patience is set to 50 during training, and the initial learning rate is adjusted to 0.001. After a certain round of training, the model can reach more than 83% mAP@0.5. Figure 2 As shown;

[0098] Step 106: Model migration and deployment, deploying the pre-trained model weight file on the edge computing device, the edge computing device is Jetson Nano 8G, to achieve rapid identification of people falling into the water;

[0099] Step 107: The heterogeneous unmanned system dynamically adjusts the search strategy and dynamically adjusts the flight altitude of the UAV according to the confidence of the identified target. The unmanned boat maintains a certain accompanying speed and tracking distance according to the flight altitude and cruising speed of the UAV to ensure the communication quality. The expression for updating the UAV altitude is as follows:

[0100] h new =h base +α·target conf (4)

[0101] Among them, h new is the updated flight altitude, h base is to set the flight altitude, α is the adjustment coefficient, target conf is the target confidence;

[0102] In addition, based on the above-mentioned heterogeneous unmanned system surface water person recognition process design, a specific implementation example of the water person positioning is further described, and the specific steps are as follows:

[0103] The process of UAV visual positioning involves the conversion of multiple coordinate systems, such as ground coordinate system, camera coordinate system, image coordinate system and pixel coordinate system. The coordinate diagram is as follows: Figure 3 As shown, the steps of UAV visual positioning are as follows:

[0104] Step 201: First, the UAV video acquisition module is calibrated to obtain the equivalent focal lengths of the horizontal and vertical axes and the coordinates of the principal point; the UAV uses RTK to obtain the altitude and longitude and latitude of the UAV during flight, and the target detection algorithm is turned on to identify the surface target and obtain the position information of the target point in the image (x u ,y u ), the coordinate system of the drone video acquisition module is as follows Figure 4 As shown, where (X u ,Y u ) The origin of the coordinate system is O u The intersection of the camera optical axis and the image plane coordinate system is located at the center of the image. u Axis, Y u The axes are parallel to the U and V axes of the image coordinate system. Assume that the physical distance between the U and V axes of each pixel in the image is dx and dy, O u The coordinates in the image coordinate system, then the coordinate conversion relationship between the imaging coordinate system and the picture coordinate system is:

[0105]

[0106] Step 202: According to the equivalent focal length dx, dy of the drone video acquisition module and the pixel information from the target point to the image principal point, the distance between the target point and the image principal point in the horizontal and vertical axes in the imaging plane can be obtained. The projection relationship of a point outside the camera on the image plane and the camera plane conforms to the pinhole imaging principle. Then the projection of a point P in space on the image plane and the camera plane is as follows: Figure 5 As shown, f is the focal length of the camera. According to the similarity relationship of triangles, the formula can be obtained as follows:

[0107]

[0108] Step 203: using the principle of similar triangles, the distance between the target point and the main point of the image in the horizontal and vertical axis directions can be obtained, and the coordinates of the target point in the video acquisition module coordinate system can be obtained;

[0109] Step 204: The coordinates of the target point in the camera coordinate system are converted to the coordinates in the earth coordinate system.

[0110] Among them, the UAV target positioning process based on single-frame images is as follows: Figure 6 As shown, the above error factors are all included in the target positioning process.

[0111] Further, in combination with the drone target positioning embodiment, the specific steps for positioning a person who falls into the water on an unmanned boat are as follows:

[0112] Step 301: The visual detection part first obtains a water surface image through a camera, and then inputs the image into the improved YOLOv8 model target detection network to obtain the confidence and prediction box coordinates of the candidate target;

[0113] Step 302: The laser radar part first processes the raw data into the form of PCD point cloud data, and then projects the point cloud data onto the image plane through the conversion matrix obtained by calibrating the internal and external parameters to achieve the conversion from high-dimensional points to low-dimensional points;

[0114] Step 303: traverse the prediction box area, count the number of point clouds projected to the image plane, calculate the ratio of the point cloud pixel area to the prediction box area, further obtain the confidence of the point cloud detection according to the zoom factor, and after obtaining the detection confidence of the two sensors, finally select the target of the person who fell into the water with high confidence by decision-level fusion;

[0115] Furthermore, the unmanned boat and the drone interact, jointly identify and locate the person who falls into the water. The positioning of the unmanned boat for the person who falls into the water is mainly in the short-range stage. The visual camera and the laser radar are fused to obtain the relative position relationship of the person who falls into the water, and the latitude and longitude coordinates of the person who falls into the water are converted through the latitude and longitude coordinates of the unmanned boat itself. The camera and laser radar data fusion detection framework is as follows: Figure 7 As shown; further, the unmanned boat receives the real-time positioning data of the UAV, fuses the target position information of the two using the Kalman filter, and obtains the fused target position information by setting the confidence threshold.

[0116] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0117] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. An interactive method for identifying and locating a person who has fallen into water, characterized in that: include: Step S1: Start the video acquisition module and search for the person who fell into the water through the person who fell into the water recognition module; Step S2: using the man-in-water target positioning module to calculate the camera coordinates of the man in the water, and combining with the RTK positioning module to convert the geodetic coordinates of the man in the water target and send them to the unmanned boat; Step S3: The unmanned boat slows down after approaching the preset range of the person falling into the water, and starts the video acquisition module of the horizontal perspective to obtain the information of the person falling into the water, uses the person falling into the water target recognition module to identify the person falling into the water, and integrates the thermal imaging camera and laser radar ranging information to further identify and locate the surface target; Step S4: Use the Kalman filter to fuse the positioning information of the person in the water provided by the UAV and the unmanned boat to obtain the precise geodetic coordinates of the person in the water.

2. The interactive method for identifying and locating a person who has fallen into water according to claim 1, characterized in that: In step S1: The implementation steps of the drowning person identification module are as follows: Establish data sets through three methods: existing public data sets, virtual simulation data sets, and self-made data sets; The public data set is collected by consulting literature to obtain a data set for identifying people falling into the water, including the SeaDroneSee data set; The virtual simulation data set provides a basis for the construction of parallel surface drowning person identification targets between the virtual world and the real world through the virtual simulation platform; The self-made dataset collects the target of people falling into the water on the spot through the video acquisition module equipped with drones and unmanned boats. Considering the richness of the dataset in different environments, the floating state and swimming posture of the person on the water are used as the falling posture and used as the image source of the dataset, and the self-made video images are preprocessed. The video acquisition module includes visible light images, thermal imaging images, and infrared images.

3. The interactive method for identifying and locating a person who has fallen into water according to claim 2, characterized in that: Preprocess the video images, divide the video containing the target of falling into water into frames, obtain each frame of the image containing the target of falling into water, remove the noise by using the median filter, use the histogram equalization to enhance the contrast, and then use the LabelImg software to label each frame of the image containing the target of falling into water, and delete the images that affect the training results if the target of falling into water is less obvious than the preset standard or the image blur is higher than the preset standard; Based on the improved YOLOv8 model pre-training, the pre-processed image dataset is resized to 640*640 as the YOLOv8 model input; by running the YOLOv8 model, the training results of each image dataset are obtained, including confidence, frame rate, and category information; by improving the YOLOv8 head network, setting the confidence threshold, and filtering the image data with low confidence, the training recognition results are obtained; Model migration and deployment: deploy the trained model weight file to the edge computing devices carried by drones and unmanned boats, and use the video acquisition modules carried by each to autonomously identify the target video images of people falling into the water captured in real time through the pre-trained YOLOv8 model, and further optimize the model parameters based on the recognition results.

4. The interactive method for identifying and locating a person who has fallen into water according to claim 1, characterized in that: In step S2: The implementation steps of the drowning person target positioning module are as follows: The video acquisition module is calibrated to obtain the equivalent focal length of the horizontal and vertical axes of the video acquisition module and the coordinates of the main point of the window; the drone uses the RTK positioning module to obtain the altitude and longitude and latitude above the ground during the flight, and turns on the target detection algorithm to identify the person falling into the water and obtain the position information of the target falling into the water in the image; The distance between the target point and the image principal point in the horizontal axis and the vertical axis direction in the imaging plane is obtained according to the equivalent focal length of the video acquisition module and the pixel information from the target point to the image principal point; The pixel distance between the target point and the main point of the window in the horizontal and vertical directions is calculated using the principle of similar triangles, and the coordinates of the target point in the coordinate system of the video acquisition module are obtained; The coordinates of the target point in the video acquisition module coordinate system are converted into the coordinates in the earth coordinate system.

5. The interactive method for identifying and locating a person who has fallen into water according to claim 4, characterized in that: The unmanned boat fuses the laser radar data and the video acquisition module data through the Kalman filter algorithm to obtain the relative position information of the target for identifying the person in the water, and then converts it through the positioning data of the boat itself to obtain the geodetic coordinates of the target for identifying the person in the water; The target location data located by the UAV and the unmanned boat are fused using the triangulation positioning algorithm to obtain the positioning data of the target falling into the water; Dynamically adjust the search strategy: The UAV dynamically adjusts the flight altitude according to the confidence of the identified target, and the unmanned boat maintains the preset escort speed and tracking distance according to the flight altitude and cruising speed of the UAV to ensure communication quality.

6. An interactive water surface drowning person identification and positioning system, characterized in that: The method for interactively identifying and locating a person falling into water according to any one of claims 1 to 5 comprises: Video acquisition module: The acquisition module obtains the information of the person who falls into the water, and cooperates with the person who falls into the water identification module to search for the person who falls into the water; Falling-in-water target positioning module: Calculate the camera coordinates of the falling-in-water person High-precision RTK positioning module: Combined with the man-in-water positioning module, the geodetic coordinates of the man-in-water target are calculated and sent to the unmanned boat; Drowned person target recognition module: identifies drowned persons and integrates thermal imaging camera and lidar ranging information to further identify and locate surface targets.

7. The interactive drowning person identification and positioning system according to claim 6, characterized in that: In the drowning person identification module: Establish data sets through three methods: existing public data sets, virtual simulation data sets, and self-made data sets; The public data set is collected by consulting literature to obtain a data set for identifying people falling into the water, including the SeaDroneSee data set; The virtual simulation data set provides a basis for the construction of parallel surface drowning person identification targets between the virtual world and the real world through the virtual simulation platform; The self-made dataset collects the target of people falling into the water on the spot through the video acquisition module equipped with drones and unmanned boats. Considering the richness of the dataset in different environments, the floating state and swimming posture of the person on the water are used as the falling posture and used as the image source of the dataset, and the self-made video images are preprocessed. The video acquisition module includes visible light images, thermal imaging images, and infrared images.

8. The interactive drowning person identification and positioning system according to claim 7, characterized in that: Preprocess the video images, divide the video containing the target of falling into water into frames, obtain each frame of the image containing the target of falling into water, remove the noise by using the median filter, use the histogram equalization to enhance the contrast, and then use the LabelImg software to label each frame of the image containing the target of falling into water, and delete the images that affect the training results if the target of falling into water is less obvious than the preset standard or the image blur is higher than the preset standard; Based on the improved YOLOv8 model pre-training, the pre-processed image dataset is resized to 640*640 as the YOLOv8 model input; by running the YOLOv8 model, the training results of each image dataset are obtained, including confidence, frame rate, and category information; by improving the YOLOv8 head network, setting the confidence threshold, and filtering the image data with low confidence, the training recognition results are obtained; Model migration and deployment: deploy the trained model weight file to the edge computing devices carried by drones and unmanned boats, and use the video acquisition modules carried by each to autonomously identify the target video images of people falling into the water captured in real time through the pre-trained YOLOv8 model, and further optimize the model parameters based on the recognition results.

9. The interactive drowning person identification and positioning system according to claim 6, characterized in that: In the drowning person target positioning module: The video acquisition module is calibrated to obtain the equivalent focal length of the horizontal and vertical axes of the video acquisition module and the coordinates of the main point of the window; the drone uses RTK to obtain the altitude and longitude and latitude above the ground during the flight, and the target detection algorithm is used to identify the person falling into the water and obtain the position information of the target falling into the water in the image; The distance between the target point and the image principal point in the horizontal axis and the vertical axis direction in the imaging plane is obtained according to the equivalent focal length of the video acquisition module and the pixel information from the target point to the image principal point; The pixel distance between the target point and the main point of the window in the horizontal and vertical directions is calculated using the principle of similar triangles, and the coordinates of the target point in the coordinate system of the video acquisition module are obtained; The coordinates of the target point in the video acquisition module coordinate system are converted into the coordinates in the earth coordinate system.

10. The interactive drowning person identification and positioning system according to claim 9, characterized in that: The unmanned boat fuses the laser radar data and the video acquisition module data through the Kalman filter algorithm to obtain the relative position information of the target for identifying the person in the water, and then converts it through the positioning data of the boat itself to obtain the geodetic coordinates of the target for identifying the person in the water; The target location data located by the UAV and the unmanned boat are fused using the triangulation positioning algorithm to obtain the positioning data of the target falling into the water; Dynamically adjust the search strategy: The UAV dynamically adjusts the flight altitude according to the confidence of the identified target, and the unmanned boat maintains the preset escort speed and tracking distance according to the flight altitude and cruising speed of the UAV to ensure communication quality.

Images