Method and system for identifying a fallen person based on a water search and rescue unmanned aerial vehicle, and medium

Abstract

The application discloses a kind of based on water search and rescue unmanned aerial vehicle's method, system and medium for identifying fallen personnel, belong to water search and rescue field, include: training phase: training set is used to train fallen personnel identification network, training sample in the training set is the picture data of open past fallen personnel, label is the position information of fallen personnel;The fallen personnel identification network is improved YOLOv6 network, including: skeleton, neck and detection head;The feature fusion module includes downsampling operation, splicing operation and 1*1 convolution operation;Application stage: the sample to be measured is input to the trained fallen personnel identification network, and the position information of fallen personnel is obtained.Simultaneously also provided corresponding data enhancement processing mode and the mode of filtering to unreasonable detection result.The application can improve the precision and speed of fallen personnel identification, is conducive to quickly and accurately complete search and rescue work, improve the survival probability of fallen personnel, reduce life and property loss.

Description

Technical Field

[0001] This invention belongs to the field of maritime search and rescue, and more specifically, relates to a method, system, and medium for identifying people who have fallen into the water based on a maritime search and rescue drone. Background Technology

[0002] Drones, with their high maneuverability and wide field of view, have been widely used in the field of water search and rescue. However, when search and rescue organizations use conventional drones to search for people who have fallen into the water, rescuers still need to manually search for the victims using their eyes on the real-time images transmitted by the drones. Considering various environmental factors and the possibility of human misjudgment and negligence, the efficiency and reliability of this method are not ideal.

[0003] Drones with autonomous detection capabilities can provide auxiliary positioning information during manual search and rescue operations, further improving efficiency and supplementing human resources. Simultaneously, by combining intelligent cruise technology, drones with strong autonomous detection capabilities can automatically complete water search and rescue operations, not only assisting human search teams but also helping with some waterborne inspection work, further reducing labor costs. However, when drones conduct search and rescue operations at a certain altitude, the identification of people in the water falls under the category of small target detection. Existing neural network detection methods, such as YOLOX-Tiny, YOLOv6-N, and YOLOv8-N, have limited accuracy and speed in detecting small targets, making it difficult to complete search and rescue operations quickly and accurately. Therefore, developing more accurate and efficient detection methods to complete search and rescue operations more quickly and accurately is of great significance for improving the survival rate of people in the water and reducing loss of life and property. Summary of the Invention

[0004] In view of the shortcomings of existing technologies and the need for improvement, this invention provides a method, system and medium for identifying people who have fallen into the water based on a waterborne search and rescue drone, with the aim of improving the accuracy and speed of identifying people who have fallen into the water.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a method for identifying persons who have fallen into the water based on a waterborne search and rescue drone is provided, comprising:

[0006] Training phase: The drowning person recognition network is trained using a training set. The training samples in the training set are publicly available images of drowning persons from the past, and the labels are the location information of the drowning persons. The drowning person recognition network is an improved YOLOv6 network, including: skeleton, neck and detection head.

[0007] The skeleton is used to extract features of different dimensions from the input training samples;

[0008] The neck includes a feature fusion module UC, which includes downsampling, concatenation, and 1x1 convolution operations; the low-dimensional features c adjacent to the current dimension. i-1 After downsampling, compared with the feature c of the current dimension i Perform a splicing operation, followed by a 1*1 convolution operation to achieve the fusion of low- and medium-dimensional features;

[0009] The detection head is used to detect the low- and medium-dimensional features and high-dimensional features after feature integration and fusion to obtain the predicted location information of the person who fell into the water.

[0010] Application phase: The test sample is input into the trained drowning person recognition network to obtain the location information of the drowning person; wherein, the test sample is the corresponding image data extracted from the video frames of the drowning person captured by the drone.

[0011] Furthermore, before the training samples are input into the skeleton, the data augmentation process is performed on the training samples, including the following steps:

[0012] For any training sample image, three images are randomly selected from the same training batch as auxiliary images, resulting in four images;

[0013] Generate a blank image of the same size as the training sample image as the background; randomly generate a point in an area more than a quarter of the side length away from the edge of the blank image as the stitching center;

[0014] The four images are randomly scaled according to a specific magnification, and stitched together in a random order to the upper left, upper right, lower left, and lower right areas of the blank image, using the stitching center as the stitching point. The portion exceeding the background area of ​​the blank image is cropped out, and then HSV enhancement is performed to obtain the enhanced image and corresponding calibration data. The calibration data is the location information of the person who fell into the water.

[0015] Furthermore, the calibration data corresponding to the enhanced image includes:

[0016] Based on the corresponding positional relationship between the enhanced image and the four images, the calibration data corresponding to the enhanced image is obtained using the calibration data corresponding to the four images.

[0017] Furthermore, it also includes filtering unreasonable detection results in the trained drowning person identification network, the filtering steps of which include:

[0018] S11. Obtain the shooting height corresponding to the image input to the trained drowning person recognition network;

[0019] S12. Calculate the possible pixel size of the person who fell into the water in the image based on the shooting height;

[0020] S13. Compare the difference between the pixel size and the predicted detection box size. If the difference is greater than a set threshold range, the predicted detection box is considered an unreasonable detection result, and the unreasonable detection result is deleted.

[0021] Furthermore, during the training phase, with the goal of minimizing the difference loss between the predicted location information of the person who fell into the water and the label, the person identification network is trained and its parameters are adjusted in reverse to bring the loss to converge.

[0022] Furthermore, the loss function L box for:

[0023]

[0024] Where IoU represents the intersection-union ratio between the predicted detection box and the detection box corresponding to the input sample label; Δ represents the distance loss between the predicted detection box and the detection box corresponding to the sample label; and Ω represents the shape loss between the predicted detection box and the detection box corresponding to the sample label.

[0025] Furthermore, the distance loss Δ and the shape loss Ω are respectively:

[0026]

[0027]

[0028] in:

[0029]

[0030]

[0031] In the formula, θ represents the degree of attention paid to shape loss; The coordinates of the center of the detection box corresponding to the sample label; c represents the center coordinates of the predicted detection box. w c h Λ represents the width and height of the minimum bounding rectangle of the detection box corresponding to the sample label; Λ represents the angle loss; w gt h gt The width and height of the detection box corresponding to the sample label are represented; w and h represent the width and height of the predicted detection box, respectively.

[0032] Furthermore, before the training samples are input into the skeleton, the training samples are sliced ​​to adapt them to the input of the drowning person identification network.

[0033] According to another aspect of the present invention, a system for identifying persons who have fallen into the water based on a waterborne search and rescue drone is provided, including a computer-readable storage medium and a processor;

[0034] The computer-readable storage medium is used to store executable instructions;

[0035] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the first aspects.

[0036] According to another aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the first aspects.

[0037] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0038] (1) The drowning person identification network designed in this invention uses a feature fusion module in the neck that only performs a splicing operation on the current dimension features and the adjacent low-dimensional features after downsampling, and then performs a 1*1 convolution operation to achieve the extraction of low- and medium-dimensional features of the skeleton. Compared with the structure of existing networks that directly fuse high- and low-level information in the neck, the feature fusion module of this invention is more conducive to preserving the features of small targets and can improve the detection and identification accuracy of small targets (drowning persons). In addition, the neck directly inputs the fused low- and medium-dimensional features into the detection head for task prediction after feature integration, instead of performing a second fusion of the fused feature map, thus avoiding the fusion of low- and medium-dimensional features into high dimensions again, further improving the network's detection and identification accuracy of small targets. At the same time, this design of this invention also simplifies the structure of the neck, making the model smaller, faster, and more suitable for deployment on the embedded mobile computing platform of UAVs.

[0039] (2) Furthermore, the data augmentation method of the present invention sets a blank image identical to the training set image as the background, generates a stitched image on the blank image, and then directly crops the part that exceeds the background area of ​​the blank image to obtain the augmented image. This avoids the existing method of directly scaling the augmentation, which will make the small target smaller and is not conducive to the detection and recognition of the drowning person identification network. The data augmentation method of the present invention retains the original size information of the small target while augmenting the data, further improving the network's detection and recognition accuracy of the small target.

[0040] (3) The method of the present invention improves the detection accuracy by filtering unreasonable detection results in the trained drowning person identification network.

[0041] (4) The loss function designed in this invention considers the cross-union ratio, distance loss and shape loss between the predicted detection box and the detection box corresponding to the sample label. When the cross-union ratio result is invalid, the loss can also be calculated by distance loss and shape loss, which improves the accuracy of network training. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the drowning person identification network model of the present invention.

[0043] Figure 2 This is a schematic diagram of the feature fusion module structure of the present invention.

[0044] Figure 3 This is a flowchart of the identification method in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0046] like Figures 1-2 As shown, the method for identifying people who have fallen into the water based on a water rescue drone of the present invention mainly includes:

[0047] Training phase:

[0048] The training set is used to train the drowning person recognition network. The training samples in the training set are publicly available images of drowning people in the past, and the labels are the location information of the drowning people. The drowning person recognition network is an improved YOLOv6 network, which includes: skeleton, neck and head detection.

[0049] The backbone is used to extract features of different dimensions from the input training samples;

[0050] The neck module includes a feature fusion module UC, which comprises downsampling, concatenation, and 1x1 convolution operations; and low-dimensional features c adjacent to the current dimension. i-1 After downsampling, compared with the feature c of the current dimension i Perform a splicing operation, followed by a 1*1 convolution operation to achieve the fusion of low- and medium-dimensional features;

[0051] The detection head is used to detect the low- and medium-dimensional features and high-dimensional features after feature integration, and obtain the predicted location information of the person who fell into the water; specifically, the RepBlock module is used to integrate the low- and medium-dimensional features after fusion.

[0052] Application phase:

[0053] The test samples are input into a trained network for recognizing people who have fallen into the water to obtain the location information of the people who have fallen into the water; the test samples are corresponding images extracted from video frames of people who have fallen into the water taken by drones.

[0054] Specifically, before inputting the training samples into the backbone, the process includes slicing the training samples to convert high-resolution image slices into a size suitable for input to the drowning person recognition network. Simultaneously, based on the relationship between the original image and the sliced ​​images, calibration data corresponding to the slices is generated using the original calibration data. Compared to compressing the original image, slicing is more effective at preserving image details, thus improving detection performance.

[0055] In this embodiment of the invention, the slicing function provided by the open-source library SAHI is used, which can efficiently perform slicing operations with a set overlap rate and output resolution. Simultaneously, based on the provided original calibration data, SAHI can automatically convert the original calibration data into the corresponding calibration data after image segmentation. Furthermore, before and after slicing with SAHI, corresponding data format conversions are required. Before slicing, the calibration data needs to be converted from YOLO format to COCO format; after SAHI processing, the new calibration data is then converted from COCO format to YOLO format.

[0056] As a further design of the present invention, before the training samples are input into the backbone, the method further includes enhancing the training samples, including the following steps:

[0057] For any training sample image, three images are randomly selected from the same training batch as auxiliary images, resulting in four images;

[0058] Generate a blank image of the same size as the training set images as the background; randomly generate a point in a region more than a quarter of the side length away from the edge of the blank image as the center of the stitching;

[0059] Four images are randomly scaled according to a specific ratio, and then randomly stitched into the upper left, upper right, lower left, and lower right areas of a blank image, with the stitching center as the stitching point, to obtain a stitched image. At the same time, the parts that exceed the background area of ​​the blank image are cropped out, and then HSV enhancement is performed to obtain the data-enhanced image.

[0060] Based on the corresponding positional relationship between the enhanced image and the original image (the four images used for stitching), the calibration data of the enhanced image is obtained using the calibration data corresponding to the original image. This calibration data contains the location information of the person who fell into the water.

[0061] Specifically, during training, the goal is to minimize the difference loss between the predicted location information of the person who fell into the water and the label. The network for identifying the person who fell into the water is trained, and its parameters are adjusted in reverse to bring the loss to convergence. The loss function L constructed in this invention... box for:

[0062]

[0063] Wherein, IoU represents the intersection-union ratio of the predicted location information of the person who fell into the water (the predicted detection box) and the corresponding label of the input training sample (the detection box corresponding to the sample label); Δ represents the distance loss between the predicted detection box and the detection box corresponding to the sample label; and Ω represents the shape loss between the predicted detection box and the detection box corresponding to the sample label.

[0064] In this embodiment of the invention, Δ and Ω are respectively:

[0065]

[0066]

[0067] in:

[0068]

[0069]

[0070] In the formula, θ represents the degree of attention paid to shape loss, which is set according to empirical values; The coordinates of the center of the detection box corresponding to the sample label; c represents the center coordinates of the predicted detection box. w c h Λ represents the width and height of the minimum bounding rectangle of the detection box corresponding to the sample label; Λ represents the angle loss; w gt h gt The width and height of the detection box corresponding to the sample label are represented; w and h represent the width and height of the predicted detection box, respectively.

[0071]

[0072] in:

[0073]

[0074]

[0075] σ is the distance between the center point of the predicted detection box and the center point of the detection box corresponding to the sample label; c h This represents the height difference between the center point of the predicted detection box and the center point of the detection box corresponding to the sample label.

[0076] As a further design of the present invention, it also includes: using data from the test set to input into the trained drowning person identification network for inference, and filtering unreasonable detection results to improve detection accuracy. The filtering method specifically includes:

[0077] S11. Read the filename of each test image to obtain the shooting height corresponding to the image; the image filename contains the shooting height corresponding to the image.

[0078] S12. Calculate the possible pixel size of the person falling into the water in the corresponding image based on the shooting height of the image. For example, for the same camera, if the maximum size of the person falling into the water detected at a height of 10m is no more than 100*100 pixels and the minimum is no less than 10*10 pixels, then the maximum size of the person falling into the water detected at a height of 20m is no more than 50*50 pixels and the minimum is no less than 5*5 pixels. And so on, the possible pixel size of the person falling into the water in the image at any height can be calculated.

[0079] S13. Compare the possible pixel size with the detection box size predicted by the drowning person identification network. If the difference is greater than the set threshold range (i.e., the predicted detection box is much larger or much smaller than the calculated possible pixel size), the predicted detection box is considered an unreasonable detection result and the unreasonable detection result is deleted.

[0080] Specifically, one specific implementation process in an embodiment of the present invention is as follows: Figure 3 As shown, S1-S4 correspond to the training phase, S5 corresponds to the test filtering phase, and S6 corresponds to the application phase. In practical applications, the training phases corresponding to S1-S4 only need to be trained once, and the model can be directly applied multiple times after successful training. A specific implementation process in an embodiment of the present invention includes:

[0081] Training phase: Collect publicly available videos of people falling into the water taken by drones in the past, and extract the corresponding image data from the videos to form a dataset. Divide the dataset into training set, validation set and test set. After data augmentation by slicing the training set, input it into the falling person recognition network for training, and continuously iterate and optimize. Calculate the accuracy of the model on the validation set after different iteration cycles, and select the model with the highest accuracy as the result.

[0082] Test filtering phase: Input the test set data into the trained model for inference, filter unreasonable detection results through the filtering algorithm, and calculate the accuracy.

[0083] Application phase: Implement the autonomous detection system on drones with suitable hardware and put it into practical use.

[0084] Because edge hardware devices on drones do not have the flexibility of general computers in terms of operating systems, drivers, computing resources, memory, testing and tuning, the network model designed in this invention is smaller in scale, simpler in structure, faster in speed, and more suitable for being mounted on the embedded mobile computing platform of drones.

[0085] Under the same experimental environment, the prediction performance of the drowning person identification network of the present invention was compared with that of different existing network structures in the embodiments of the present invention, as shown in Table 1 below:

[0086] Table 1 Performance comparison of different network models and the network model of this invention.

[0087] Model mAP@0.5 (%) mAP(%) Parameters Average inference time YOLOv5-N 28.1 9.5 1.9M 1.8ms YOLOX-Tiny 23.7 8.1 5.1M 6.5ms YOLOv6-N 35.9 13.5 4.6M 2.4ms YOLOv7-Tiny 38.2 14.1 6.2M 2.2ms YOLOv8-N 32.3 11.8 3.2M 2.5ms Embodiments of the present invention 43.2 15.7 3.6M 2.0ms

[0088] In Table 1, mAP represents the average accuracy calculated for a set of IoU thresholds (from 0.5 to 0.95, with a step size of 0.05); mAP@0.5 represents the average accuracy calculated for a single IoU threshold (i.e., 0.5).

[0089] It can be seen that, under the same experimental conditions, the mAP@0.5 and mAP values ​​in the embodiments of the present invention are both higher, and the average inference time is also shorter.

[0090] According to another aspect of the present invention, a system for identifying persons who have fallen into the water based on a waterborne search and rescue drone is provided, including a computer-readable storage medium and a processor;

[0091] Computer-readable storage media are used to store executable instructions;

[0092] The processor is used to read executable instructions stored in a computer-readable storage medium and execute the steps corresponding to the method for identifying people who have fallen into the water based on a waterborne search and rescue drone in the above embodiments.

[0093] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps corresponding to the method for identifying drowning persons based on a waterborne search and rescue drone as described in the above embodiments.

[0094] The drowning person recognition network designed in this invention uses a feature fusion module (UC) in the neck that only concatenates the current dimension features with the downsampled adjacent low-dimensional features before performing a 1*1 convolution operation. This achieves the extraction of mid-to-low-dimensional features from the backbone. Compared to existing neck structures that directly fuse mid-to-high-level and low-level information, the feature fusion module (UC) of this invention is more conducive to preserving the features of small targets, thus improving the detection and recognition accuracy of small targets (drowning persons). Furthermore, in the neck, the fused mid-to-low-dimensional features are directly input into the head for task prediction after feature integration, rather than undergoing a second fusion of the fused feature maps. This avoids fusing mid-to-low-dimensional features back into high dimensions, further improving the network's detection and recognition accuracy for small targets. This design also simplifies the neck structure, making the model smaller, faster, and more suitable for deployment on embedded mobile computing platforms of drones.

[0095] The data augmentation method of this invention sets a blank image identical to the training set images as the background, generates a stitched image on the blank image, and then directly crops the part that exceeds the background area of ​​the blank image to obtain the augmented image. This method avoids the existing method of directly scaling the augmentation, which would shrink small targets even smaller, making it difficult for the network to detect and identify people falling into the water. The data augmentation method of this invention preserves the original size of small targets while augmenting the data, further improving the network's detection and recognition accuracy of small targets.

[0096] The loss function designed in this invention considers the cross-union ratio, distance loss, and shape loss between the predicted detection box and the detection box corresponding to the sample label. When the cross-union ratio result is invalid, the loss can also be calculated through distance loss and shape loss, thereby improving the accuracy of network training.

[0097] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying a fallen person based on a water search and rescue unmanned aerial vehicle, characterized in that, The method comprises the following steps: A training phase: a fallen person recognition network is trained by using a training set, wherein training samples in the training set are open previous fallen person picture data, and labels are position information of the fallen person; The fallen person recognition network is an improved YOLOv6 network, which comprises a skeleton, a neck and a detection head; The skeleton is used to extract different dimension features of an input training sample; The neck includes a feature fusion module UC, which includes a down-sampling operation, a splicing operation and a 1*1 convolution operation; the low-dimensional feature c i-1 After the down-sampling operation, the feature c i The splicing operation is performed, and after splicing, the 1*1 convolution operation is performed to realize the fusion of the medium and low-dimensional features. The detection head is used to detect low and high dimension features after fusion and feature integration to obtain predicted position information of the fallen person; An application phase: a to-be-tested sample is input into the trained fallen person recognition network to obtain position information of the fallen person, wherein the to-be-tested sample is corresponding picture data cut from a fallen person video frame shot by a drone; In the training phase, a difference loss between the predicted position information of the fallen person and the label is minimized to train the fallen person recognition network and reversely adjust parameters of the fallen person recognition network to make the loss converge; Loss function is: wherein, represents an intersection over union of the predicted bounding box and the bounding box corresponding to the input sample label; represents a distance loss between the predicted bounding box and the bounding box corresponding to the sample label; represents a shape loss between the predicted bounding box and the bounding box corresponding to the sample label; the distance loss and the shape loss respectively. Before the training sample is input into the skeleton, the training sample is subjected to data enhancement processing, which comprises the following steps: In the formula, represents the degree of attention to shape loss control; , is the center coordinate of the detection box corresponding to the sample label; , is the center coordinate of the predicted detection box; , is the width and height of the minimum circumscribed rectangle of the detection box corresponding to the sample label; and Λ is the angle loss. , represents the width and height of the detection box corresponding to the sample label; respectively represent the width and height of the predicted detection box.

2. The method of claim 1, wherein, For any training sample picture, three pictures are randomly selected as auxiliary pictures in the same training batch to obtain four pictures; A blank picture with the same size as the training sample picture is generated as a background, and a point is randomly generated as a splicing center at a region with a distance from the blank picture edge exceeding one fourth of the length; The four pictures are randomly scaled by a specific ratio, spliced into the upper left, upper right, lower left and lower right regions of the blank picture as the splicing center, and the parts exceeding the background region of the blank picture are cut off, and then subjected to HSV enhancement to obtain an enhanced picture and corresponding calibration data; the calibration data is position information of the fallen person. The calibration data corresponding to the enhanced picture comprises:

3. The method of claim 2, wherein, According to the corresponding position relationship between the enhanced picture and the four pictures, the calibration data corresponding to the enhanced picture is obtained by using the calibration data corresponding to the four pictures. The trained fallen person recognition network is further filtered for unreasonable detection results, and the filtering steps comprise:

4. The method of claim 1, wherein, S11, acquiring a shooting height corresponding to a picture input into the trained fallen person recognition network; S12, calculating a possible pixel size of the fallen person in the picture according to the shooting height; S13, comparing the pixel size with a predicted bounding box size, and if the difference is greater than a set threshold range, considering that the predicted bounding box is an unreasonable detection result, and deleting the unreasonable detection result. Before the training sample is input into the skeleton, the training sample is further subjected to slicing processing to adapt to the input of the fallen person recognition network.

5. The method of claim 1, wherein, The system comprises a computer readable storage medium and a processor; 6. A water-based search and rescue drone-based fallen person identification system, characterized by, The computer readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer readable storage medium to execute the method in any one of claims 1-5. ​ 7. A computer readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by the processor, implements the method of any one of claims 1-5.

Images