A lightweight UAV image-based algorithm for detecting people falling into water, based on YOLOv7.

Abstract

This invention belongs to the field of deep learning object detection technology, and specifically provides a lightweight drone image-based algorithm for detecting people falling into water, based on YOLOv7. The algorithm includes the following steps: S1, collecting a suitable number of images of swimming and people falling into water to create a dataset; S2, replacing the YOLOv7 backbone network with MobileNetV4; S3, reconstructing the SPPC / SPC structure of YOLOv7 and introducing a CA attention mechanism and a Wise-IoU loss function; S4, conducting ablation experiments on a self-made dataset; S5, comparing the improved algorithm with other object detection algorithms on the self-made dataset. The algorithm proposed in this invention significantly reduces the number of parameters and computational cost of the original algorithm while only sacrificing a small amount of detection accuracy, and greatly improves the detection speed. By making the original algorithm lightweight, it is easier to deploy on edge devices with limited computing power, such as drones.

Description

Technical Field

[0001] This invention belongs to the field of deep learning target detection technology, specifically a lightweight drone image-based algorithm for detecting people falling into water based on YOLOv7. Background Technology

[0002] According to the World Health Organization, at least 236,000 people die from drowning worldwide each year, accounting for 7% of all unintentional injury-related deaths globally. These figures are not only heartbreaking but also highlight the urgent need to improve water rescue techniques. In the field of water rescue, quickly and effectively detecting people in the water is crucial for increasing rescue rates. Traditional rescue methods are limited by human resources and natural conditions, failing to meet the demands for rapid response. With the development of drone technology and deep learning, drone-based target detection offers new perspectives and possibilities for rescue operations.

[0003] Object detection technology, as one of the core tasks in the field of computer vision, has undergone revolutionary changes in recent years with the rise of deep learning. Currently, deep learning-based object detection algorithms are mainly divided into two categories: two-stage object detection algorithms, such as R-CNN, Faster R-CNN, Mask R-CNN, and SPPNet. These algorithms have high detection accuracy but are relatively slow. The other category is single-stage object detection algorithms, such as SSD and the YOLO series. These algorithms have fast detection speed but lower detection accuracy than two-stage algorithms.

[0004] Currently, for detection tasks from the perspective of UAVs, Zhao Hangyue et al. added detection heads to reduce the negative impact of excessive target scale variance; Jawaharlalnehru et al. used the K-means clustering algorithm to determine the optimal anchor box size, making it more effective in detecting targets at multiple scales; Huang Shuqin et al. proposed a scale-aware weighted loss function to dynamically assign weights to targets, thereby strengthening the contribution of small-scale targets to the loss; Willy et al. introduced SwimTransformer into DetectoRS to enhance the network's ability to capture and process global information, but the model's computational cost is too high, making it difficult to deploy on edge devices. Currently, research on detection algorithms from the perspective of UAVs is increasing, and detection accuracy is continuously improving, but the demand for real-time detection has not yet been fully met.

[0005] Model deployment is a crucial consideration in algorithm research. To accelerate model inference, numerous scholars have conducted extensive research and achieved significant progress. Xiao Jin et al. designed an adaptive perceptual spatial pyramid module, which accelerates inference and reduces the number of model parameters by dynamically adjusting decoder parameters. Zhou Quan et al. proposed a dual-path lightweight network whose computational complexity is linearly related to the input feature resolution. By processing features of different resolutions separately, it saves substantial computational costs. Guo Jinyang et al. proposed a multi-dimensional pruning framework that compresses the network along multiple dimensions in an end-to-end manner, improving model efficiency. While these lightweight designs simplify the model and improve efficiency, they significantly impact detection accuracy.

[0006] Therefore, this invention proposes a lightweight UAV image-based algorithm for detecting people falling into water based on YOLOv7 to solve the above problems. Summary of the Invention

[0007] (a) Technical problems to be solved

[0008] To address the shortcomings of existing technologies, this invention provides a lightweight UAV image-based algorithm for detecting people falling into water, based on YOLOv7, which solves the problems mentioned in the background section.

[0009] (II) Technical Solution

[0010] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0011] A lightweight UAV image-based algorithm for detecting people falling into water, based on YOLOv7, includes the following steps:

[0012] S1. Collect a suitable number of images of swimmers and people who have fallen into the water, and create a dataset;

[0013] S2. Replace the YOLOv7 backbone network with MobileNetV4;

[0014] S3. Reconstruct the SPPCSPC structure of YOLOv7 and introduce the CA attention mechanism and Wise-IoU loss function;

[0015] S4. Conduct ablation experiments on a self-made dataset;

[0016] S5. Compare the improved algorithm with other object detection algorithms on a self-made dataset.

[0017] Furthermore, the specific process of S1 involves collecting swimming and drowning images with appropriate height and resolution from the internet, collecting images of different scenes and various swimming styles as much as possible, and then using Labelme software to label the images after collection. The dataset includes two label categories: swimmers and drowning victims, and the scenes mainly include oceans, rivers, and ponds.

[0018] Furthermore, MobileNetV4 in S2 is a new generation of lightweight detection network proposed by Qin Danfeng et al. This network introduces a general inverted bottleneck block, which is an adjustable module suitable for efficient network design. It can adapt to various optimization objectives without increasing the complexity of network search. This module is an improvement on the inverted bottleneck block in MobileNetV2.

[0019] Furthermore, in S3, the SPPCSPC structure is improved by referencing the depthwise separable convolution in MobileNetV4 and the SPPELAN structure in YOLOv9 as follows:

[0020] (1) Replace the 3*3 convolution with 3*3 depthwise convolution and 1*1 pointwise convolution to significantly reduce the number of convolution parameters;

[0021] (2) The pooling layer structure was redesigned, and the pooling window size was changed from (5, 9, 13) to three serial pooling layers with a window size of 5 to improve the calculation speed.

[0022] (3) Replace the SiLU function in the first and last 1*1 convolutional layers with the ReLU function to improve the model detection accuracy. The position and number of ReLU functions are based on experiments.

[0023] The CA attention mechanism in S3 encodes channel relationships and long-term dependencies through precise location information. The specific operation is divided into two steps, a and b, of Coordinate information embedding and Coordinate Attention generation.

[0024] a. Coordinate information embedding

[0025] Global pooling is typically used for global encoding of spatial information in channel attention encoding. However, because it compresses global spatial information into channel descriptors, it makes it difficult to preserve positional information. To enable the attention module to capture remote spatial interactions with precise positional information, global pooling is decomposed into a one-to-one feature encoding operation according to the following formula:

[0026]

[0027] Specifically, given an input X, each channel is first encoded along the horizontal and vertical coordinates using a pooling kernel of size (H, 1) or (1, W). Therefore, the output of the c-th channel with height h can be expressed as:

[0028]

[0029] Similarly, the output of the c-th channel with width w can be written as:

[0030]

[0031] The two transformations described above aggregate features along two spatial directions respectively, resulting in a pair of direction-aware feature maps;

[0032] b. Coordinate Attention Generation

[0033] After the transformation in information embedding, this part performs a concatenation operation on the above transformation, and then uses a 1*1 convolution transformation function F1 to transform it:

[0034] f=δ(F1([z h ,z w ]))

[0035] In the formula, [·, ·] represents the concatenation operation along the spatial dimension, δ is the nonlinear activation function, f is the intermediate feature map that encodes spatial information in the horizontal and vertical directions, and r is the reduction rate used to control the size of the SE block. Then, f is decomposed into two separate tensors f along the spatial dimension. h and f w Using two other 1x1 convolution transformations F h and F w f respectively h and f w Transforming the input X into a tensor with the same number of channels yields:

[0036] g h =σ(F h (f h ))

[0037] g w =σ(F w (f w ))

[0038] Here, σ is the sigmoid activation function. To reduce model complexity and computational cost, an appropriate reduction ratio r (e.g., 32) is typically used to reduce the number of channels in f, and then the output g is... h and g wExtending this further, and using these as attention weights respectively, the output Y of the CA attention mechanism can finally be written as:

[0039]

[0040] The Wise-IoU loss function in S3 is defined as follows:

[0041] S u =wh+w gt h gt -W i H i

[0042]

[0043] L WIoU =R WIoU L IoU

[0044] Among them W g H g This represents the width and height of the minimum bounding box, to prevent R from being displayed. wiou The gradient W that hinders convergence is generated. g and H g Separate from the computation graph (the superscript * indicates this operation).

[0045] Furthermore, in step S4, to verify the effectiveness of the improvements to each module, each module was tested individually to validate the effectiveness of the improvements. The same experimental environment was used, and frame rate (FPS), parameter quantity (Params), floating-point operation count (GFLOPs), and average precision (mAP) were used as evaluation metrics to assess the effectiveness of the proposed improvement measures. Frame rate refers to the number of image refresh frames, parameter quantity is in megabytes (M), floating-point operation count is in gigabytes (G), and average precision (mAP) refers to the average accuracy of detection across all categories. The calculation formula is as follows:

[0046]

[0047] Where TP represents the number of positive samples predicted as positive samples, FP represents the number of negative samples predicted as positive samples, FN represents the number of positive samples predicted as negative samples, c represents the total number of object detection categories, and P(R) is a curve plotted with recall on the X-axis and accuracy on the Y-axis. The area enclosed by the curve and the coordinate axes is the average accuracy. Furthermore, when reconstructing the SPPCSPC structure, the convolutional layer and pooling layer structures were improved, and comparative experiments were conducted on a model with a replaced backbone network.

[0048] Furthermore, in S5, the superiority of this algorithm is demonstrated by comparing it with classic algorithms such as SSD, Faster R-CNN, and other YOLO algorithms.

[0049] (III) Beneficial Effects

[0050] Compared with existing technologies, this invention provides a lightweight UAV image-based algorithm for detecting people falling into water based on YOLOv7, which has the following advantages:

[0051] 1. The algorithm proposed in this invention significantly improves the detection speed while slightly reducing the detection accuracy, and also greatly reduces the number of model parameters and computational load.

[0052] 2. This invention makes the original algorithm easier to deploy on edge devices with limited computing power, such as drones, by making it lightweight. Attached Figure Description

[0053] Figure 1 This is a flowchart of a lightweight UAV image-based human fall-in-water detection algorithm based on YOLOv7 according to an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the improved YOLOv7 algorithm structure of this invention;

[0055] Figure 3 This is a comparison diagram of the improved SPPCSPC structure and the original SPPPCSPC structure of this invention;

[0056] Figure 4 This is a schematic diagram of the experimental environment for this invention;

[0057] Figure 5 This is a schematic diagram of the experimental parameters of the present invention;

[0058] Figure 6 , 7 The figure shows the experimental results of the improved SPPCSPC according to the present invention;

[0059] Figure 8 This is a diagram showing the ablation experiment results of this invention;

[0060] Figure 9 The image shows the comparison results of the algorithm of this invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Example

[0063] like Figure 1-9 As shown in the figure, a lightweight UAV image-based algorithm for detecting people falling into water, based on YOLOv7, is proposed in one embodiment of the present invention, including the following steps:

[0064] S1. Collect a suitable number of images of swimmers and people who have fallen into the water, and create a dataset;

[0065] This dataset collects swimming and drowning images of suitable height and resolution from the internet, aiming to gather images from different scenes and with various swimming styles. After collection, the images are labeled using Labelme software. The dataset includes two labeling categories: swimmers and drowning victims, with scenes primarily including oceans, rivers, and ponds. The dataset contains a total of 2360 images, with 1888 images in the training set, and 236 images each in the validation and test sets.

[0066] S2. Replace the YOLOv7 backbone network with MobileNetV4;

[0067] MobileNetV4 is a next-generation lightweight detection network proposed by Qin Danfeng et al. in April 2024. This network introduces a universal inverted bottleneck block, an adjustable module suitable for efficient network design that can adapt to various optimization objectives without increasing network search complexity. This module is an improvement upon the inverted bottleneck block in MobileNetV2.

[0068] S3. Reconstruct the SPPCSPC structure of YOLOv7 and introduce the CA attention mechanism and Wise-IoU loss function;

[0069] I. Reconstructing the SPPCSPC structure

[0070] Although the SPPCSPC structure performs better than the SPPF structure, its number of parameters and computational cost have also increased significantly. In order to further simplify the model and improve its performance, this invention decides to improve the SPPCSPC module.

[0071] By referencing the depthwise separable convolution in MobileNetV4 and the SPPELAN structure in YOLOv9, the SPPCSPC structure was improved as follows:

[0072] (1) Replace the 3*3 convolution with a 3*3 depthwise convolution and a 1*1 pointwise convolution to significantly reduce the number of convolution parameters.

[0073] (2) The pooling layer structure was redesigned, and the pooling window size was changed from (5, 9, 13) to three serial pooling layers with a window size of 5 to improve the calculation speed.

[0074] (3) Replace the SiLU function in the first and last 1*1 convolutional layers with the ReLU function to improve the model detection accuracy. The position and number of ReLU functions are based on experiments.

[0075] The improved module in this invention is referred to as SPPDPC, and the module structure comparison diagram is as follows. Figure 3 As shown.

[0076] II. Introducing CA Attention Mechanism

[0077] In water, the visible area of ​​swimmers and those who have fallen in is limited, and due to increased distance, some individuals may appear smaller in the image. As the number of network layers increases during detection, details and feature information of some targets may be lost, which could affect the network's recognition accuracy. To address this issue, this invention introduces a CA attention mechanism, which can effectively improve the network's detection accuracy without significantly increasing the computational burden.

[0078] The Coordinate Attention (CA) mechanism encodes channel relationships and long-term dependencies using precise location information. The specific operation consists of two steps: Coordinate information embedding and Coordinate Attention generation.

[0079] Coordinate Information Embedding: Global pooling methods are typically used for global encoding of spatial information in channel attention, but because they compress global spatial information into channel descriptors, it is difficult to preserve positional information. To enable the attention module to capture long-range spatial interactions with precise positional information, global pooling is decomposed into a one-to-one feature encoding operation according to the following formula:

[0080]

[0081] Specifically, given an input X, each channel is first encoded along the horizontal and vertical coordinates using either a pooling kernel of size (H, 1) or (1, W). Therefore, the output of the c-th channel at height h can be expressed as:

[0082]

[0083] Similarly, the output of the c-th channel with width w can be written as:

[0084]

[0085] The two transformations described above aggregate features along two spatial directions to obtain a pair of direction-aware feature maps.

[0086] Coordinate Attention Generation: After the transformation in information embedding, this part concatenates the above transformation and then uses a 1x1 convolution transformation function F1 to transform it.

[0087] f=δ(F1([z h ,z w ]))

[0088] In the formula, [·, ·] represents the concatenation operation along the spatial dimension, δ is the nonlinear activation function, and f is the intermediate feature map that encodes spatial information in the horizontal and vertical directions. Here, r is the reduction rate used to control the size of the SE block. Then, f is decomposed into two separate tensors f along the spatial dimension. h and f w Using two other 1x1 convolution transformations F h and F w f respectively h and f w Transforming the input X into a tensor with the same number of channels yields:

[0089] g h =σ(F h (f h ))

[0090] g w =σ(F w (f w ))

[0091] Here, σ is the sigmoid activation function. To reduce model complexity and computational cost, an appropriate reduction ratio r (e.g., 32) is typically used to reduce the number of channels in f. Then, the output g is... h and g w They are then expanded and used as attention weights.

[0092] Finally, the output Y of the CA attention mechanism can be written as:

[0093]

[0094] III. Introducing the Wise-IoU Loss Function

[0095] In YOLOv7, C-IoU is used as the loss function. This function comprehensively considers the overlap area between predicted boxes, the distance between center points, and the aspect ratio to calculate the regression error of the predicted boxes. However, since the training data inevitably contains low-quality samples, geometric factors (such as distance and aspect ratio) will exacerbate the penalty for low-quality samples, thereby reducing the generalization performance of the model. To mitigate the impact of geometric factors on low-quality samples, this invention chooses to introduce Wise-IoU as the loss function for YOLOv7. The definition of Wise-IoU is as follows:

[0096] S u =wh+w gt h gt -W i H i

[0097]

[0098] L WIoU =R WIoU L IoU

[0099] Among them W g H g This represents the width and height of the smallest bounding box. To prevent R... wiou The gradient W that hinders convergence is generated. g and H g Separate from the computation graph (the superscript * indicates this operation).

[0100] S4. Conduct ablation experiments on a self-made dataset;

[0101] I. Experimental Setup and Evaluation Indicators

[0102] Experimental environment and parameter settings are as follows: Figure 4 , Figure 5 As shown. This invention uses average precision (mAP), frame rate (FPS), number of parameters (Params), and number of floating-point operations (GFLOPs) as evaluation metrics to assess the effectiveness of the proposed improvement measures.

[0103] The mean accuracy refers to the average accuracy of all categories of detection, and the calculation formula is as follows:

[0104]

[0105] Where TP represents the number of positive samples predicted as positive samples, FP represents the number of negative samples predicted as positive samples, FN represents the number of positive samples predicted as negative samples, c represents the total number of categories detected, and P(R) is a curve plotted with recall on the X-axis and precision on the Y-axis. The area of ​​the graph enclosed by the curve and the coordinate axes is the average precision.

[0106] Frame rate refers to the number of frames the image is refreshed. The unit for parameter values ​​is MB. The unit for floating-point operations is GB.

[0107] II. SPPC-SPC Comparison Experiment

[0108] When reconstructing the SPPCSPC structure, we first improved the structure of the convolutional and pooling layers, and then conducted comparative experiments on a model with the backbone network replaced, such as... Figure 6 In groups (1) and (2), it can be seen that the number of parameters in the improved model decreased by 4.18M and the FPS increased slightly, showing initial success. However, we noticed that the model accuracy decreased, so we decided to improve the model accuracy. We used the ReLU activation function for adjustment and experimented with the number and position of the ReLU function in five 1*1 convolutional layers. The results are shown in […]. Figure 7 The comparative experiments were conducted in groups (3) to (7). The results show that more ReLU functions are not necessarily better. Overall, adding two ReLU functions yields the best results. To verify the optimal placement of the ReLU functions, experiments were conducted at different locations, and the results are as follows: Figure 7 As shown, overall, group (4) performed the best.

[0109] III. Ablation Experiment

[0110] To demonstrate the performance of the algorithm of this invention, we conducted ablation experiments under identical conditions. The experimental results are as follows: Figure 8 As shown, v1 represents the original YOLOv7 model, and v6 represents the improved performance.

[0111] As shown in the figures, after replacing the backbone network, mAP50 decreased by 1.6%, but FPS increased by 103%, and computational cost and parameter count decreased by 61.9% and 13.45M respectively, demonstrating the effectiveness of the lightweight model improvement. In V3, computational cost and parameter count decreased by another 8.3% and 4.18M respectively, and mAP50 increased by 0.24% while maintaining the same FPS, indicating that SPPDPC is more suitable for this network structure. In V4, mAP50 increased by 0.29%, although FPS decreased slightly, it was within an acceptable range. In V5, mAP50 increased by 0.1% while maintaining the same FPS, proving that this loss function is more suitable for this dataset. Compared to the original YOLOv7 model, the final improved model improved FPS by 97.8% while only decreasing mAP by 1.08%, and reduced parameter count and computational cost by 46.7% and 65.0% respectively.

[0112] Analysis of the above detection results reveals that the algorithm of this invention significantly reduces the number of parameters and computational load, better balancing detection accuracy and speed, and is more suitable for deployment on drones compared to the original model.

[0113] S5. Compare the improved algorithm with other object detection algorithms on a self-made dataset.

[0114] To demonstrate the effectiveness of the algorithm improvements, SSD, Faster R-CNN, and YOLO series object detection models were compared. Detection was performed on a self-made dataset under the same experimental environment and parameters. The model comparison experimental results are as follows: Figure 9 As shown.

[0115] As shown in the figure, although the two-stage Faster R-CNN uses a region proposal network, its model accuracy is still lower than that of the YOLO series, and its detection speed is slower, making it difficult to meet the requirements of real-time detection. Among the single-stage algorithms, SSD has a higher detection speed, achieving real-time detection, but the difference in speed compared to the YOLO series is not significant, and its detection accuracy is far inferior, making it difficult to meet the application needs of complex scenarios. Within the YOLO series, while YOLOv7 is slightly slower than YOLOv3 and YOLOv4 in detection speed, its detection accuracy and number of parameters are superior to both. Furthermore, YOLOv7 outperforms YOLOv5 in all metrics. Overall, YOLOv7 better balances accuracy and speed. The improved algorithm's mAP is only 1.08% lower than YOLOv7, while its detection speed is nearly doubled, and the number of parameters and computational cost are significantly reduced. Compared to other algorithms, the algorithm of this invention is more suitable for deployment on resource-constrained UAV platforms.

[0116] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lightweight UAV image-based method for detecting people falling into water, based on YOLOv7, characterized in that, Includes the following steps: S1. Collect a suitable number of images of swimmers and people who have fallen into the water, and create a dataset; S2. Replace the YOLOv7 backbone network with MobileNetV4; S3. Reconstruct the SPPCSPC structure of YOLOv7 and introduce the CA attention mechanism and Wise-IoU loss function; In S3, the SPPCSPC structure was improved by referencing the depthwise separable convolution in MobileNetV4 and the SPPELAN structure in YOLOv9 as follows: (1) Replace the 3*3 convolution with a 3*3 depthwise convolution and a 1*1 pointwise convolution, thereby significantly reducing the number of convolution parameters; (2) The pooling layer structure was redesigned, and the size of the pooling window was changed from (5, 9, 13) to three serial pooling layers with a window size of 5 to improve the calculation speed. (3) Replace the SiLU function in the first and last 1*1 convolutional layers with the ReLU function to improve the detection accuracy of the model. The position and number of ReLU functions are based on experiments. The CA attention mechanism in S3 encodes channel relationships and long-term dependencies through precise location information. The specific operation is divided into two steps, a and b, of Coordinate information embedding and Coordinate Attention generation. a. Coordinate information embedding Global pooling is typically used for global encoding of spatial information in channel attention encoding. However, because it compresses global spatial information into channel descriptors, it makes it difficult to preserve positional information. To enable the attention module to capture remote spatial interactions with precise positional information, global pooling is decomposed into a one-to-one feature encoding operation according to the following formula: ; Given an input X, each channel is first encoded along the horizontal and vertical coordinates using either a pooling kernel of size (H, 1) or (1, W). Therefore, the output of the c-th channel at height h is represented as: ; Similarly, the output of the c-th channel with width w can be written as: ; The two transformations described above aggregate features along two spatial directions respectively, resulting in a pair of direction-aware feature maps; b. Coordinate Attention Generation After the transformation in information embedding, this part concatenates the above transformation and then performs a 1x1 convolution transformation function F1 on it: ; In the formula, [·, ·] represents the concatenation operation along the spatial dimension, δ is the nonlinear activation function, f is the intermediate feature map that encodes spatial information in the horizontal and vertical directions, and then f is decomposed into two separate tensors f along the spatial dimension. h and f w Using two other 1x1 convolution transformations F h and F w f respectively h and f w Transforming the input X into a tensor with the same number of channels yields: ; ; Here, σ is the sigmoid activation function. To reduce model complexity and computational cost, an appropriate reduction ratio r is typically used to reduce the number of channels in f, and then the output g is... h and g w Expanding these values ​​and using them as attention weights, the final output Y of the CA attention mechanism is written as: ； The Wise-IoU loss function in S3 is defined as follows: ; ; ; ; Among them W g H g This represents the width and height of the smallest bounding box, to prevent... The gradient that hinders convergence is generated, * denotes W. g and H g Separate from the computation graph.

2. The lightweight UAV image-based method for detecting people falling into water based on YOLOv7 according to claim 1, characterized in that, The specific process of S1 is to collect swimming and falling-in-water images with appropriate height and resolution from the Internet, and to collect images of different scenes and various swimming styles as much as possible. After collection, the images are labeled using Labelme software. The dataset includes two label categories: swimmers and falling-in-water people, and the scenes are mainly oceans, rivers and ponds.

Images