A method for detecting pool drowning based on RTMDet

Abstract

The application discloses a pool drowning detection method based on RTMDet, constructs a heterogeneous feature extraction backbone network and a CFP-FPN module, learns through task alignment, and combines the characteristics of anchor box-based and anchor box-free auxiliary training with the aid of anchor box training to improve the effective receptive field of the model without significantly increasing the model complexity, and improve the detection accuracy of different scale swimmers in complex scenes. The CFP-FPN module proposed in the application can fully fuse the features between layers and within layers, improve the detection accuracy of the model for different scale targets, and the TAL-AAT detection head proposed fully combines the advantages of classification scores, the most accurate bounding box, anchor box-based and anchor box-free to improve the classification and positioning accuracy of the model.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically a method for detecting drowning in swimming pools based on RTMDet. Background Technology

[0002] Swimming can improve children's physical fitness, coordination, and intelligence, and promote neurological development. Therefore, more and more parents are encouraging their children to swim. However, swimming carries significant risks. Simply relying on lifeguards to monitor swimmers has limitations and cannot guarantee swimmers' safety. Researching and applying computer vision methods to assist lifeguards in monitoring swimming pools to effectively reduce the probability of accidents is of significant practical importance.

[0003] With the successful application of object detection in many fields, drowning detection based on object detection algorithms has become mainstream. Currently, deep learning-based object detection algorithms can be divided into two types: two-stage algorithms and single-stage algorithms. Two-stage algorithms first generate candidate boxes from the image, then use a learning module to extract regional features from the candidate boxes and complete the final classification and regression; representative algorithms include the R-CNN series. These algorithms have high detection accuracy but slow detection speed, making them unsuitable for real-time detection scenarios. Single-stage algorithms, on the other hand, directly perform regression and classification of the target; representative algorithms include SSD, the YOLO series, and RTMDet. These algorithms offer both fast detection speed and high detection accuracy, thus single-stage object detection algorithms have been widely used.

[0004] To detect drowning behavior, researchers both domestically and internationally have proposed numerous methods, including underwater human detection methods based on background subtraction, swimming pool drowning detection methods based on Mask R-CNN, and swimming pool drowning detection methods based on YOLOv4. However, these methods each have their own drawbacks: 1. Background modeling in background subtraction is affected by many dynamic changes in the scene, impacting subsequent target extraction and tracking, resulting in low detection accuracy in complex environments. 2. While Mask R-CNN-based detection methods offer improved accuracy, their detection speed is relatively slow. 3. Underwater detection methods based on YOLOv4 are easily obstructed by swimmers and require underwater cameras, leading to high maintenance costs. 4. Most swimming pool drowning detection methods exhibit good accuracy in simple scenarios, but their accuracy decreases in complex scenarios such as swimming pools with swimming rings and toys, reflective water surfaces, and diverse swimmer postures, making them prone to misidentification of swimmers. Summary of the Invention

[0005] The purpose of this invention is to provide a highly efficient and accurate detection system for drowning accidents in swimming pools, thereby improving the accuracy of the system in detecting drowning accidents in complex swimming pool scenarios.

[0006] To achieve the above-mentioned objectives, this invention proposes a swimming pool drowning detection method based on RTMDet, comprising the following steps:

[0007] S1) Use video capture equipment to capture pool video frames, extract images from the video frames at fixed frame intervals, label the images to obtain a pool image dataset, and divide the pool image dataset into training set images and validation set images. The training set images are used to train the network model, and the validation set images are used to evaluate the model performance and adjust the model parameters during the training process.

[0008] S2) Construct a heterogeneous feature extraction backbone network to extract multi-scale features of different sizes and resolutions from the training set images;

[0009] S3) Construct a CFP-FPN multi-scale feature pyramid, and input the multi-scale features extracted in S2) into the CFP-FPN multi-scale feature pyramid for fusion, thereby aggregating features of different resolutions to achieve effective detection of targets at different scales;

[0010] S4) Construct the TAL-AAT detection head, input the aggregated features from S3) into the TAL-AAT detection head, and combine the classification score, the most accurate bounding box, and the gradient of the model based on anchor boxes and no anchor boxes to train the network, thereby improving the accuracy of model classification and localization.

[0011] S5) Use the training set images and validation set images to iteratively train the model until the loss function converges, and save the trained model.

[0012] S6) Set up multiple cameras to monitor the swimming pool in real time and transmit the video to the drowning detection and recognition system. The system uses the swimming pool drowning detection model to output the detection results. When the system detects that drowning has occurred, it will issue an alarm and save the detected video.

[0013] Furthermore, the construction of the heterogeneous feature extraction backbone network in step S2 above includes the following steps:

[0014] S2.1: By setting the kernel size K in the backbone network to 3, 5, 7, and 9 in sequence, interference information other than the effective information of small objects can be reduced;

[0015] S2.2: Reduce model computation and speed up inference by using Pconv;

[0016] S2.3: Improve the model's attention to key features by using the SimAM attention module.

[0017] Furthermore, the construction of the CFP-FPN module in step S3 above specifically includes:

[0018] S3.1: Capturing Global Long-Term Dependencies X of Deepest Features Using a Lightweight MLP MLP ;

[0019] S3.2: The aggregation of local region features within the deepest feature layer is achieved through learnable visual center LVC, i.e., X. LVC ;

[0020] S3.3: X MLP and X LVc The pieces are stitched together along the channel dimension.

[0021] Furthermore, in step S3.1 above, the global long-term dependency X of the deepest feature is captured. MLP The calculation method is as follows:

[0022] S3.1.1: The deepest feature X5 extracted from the backbone network is processed through a Stem block containing convolutional layers, normalization layers, and ReLU activation functions to obtain feature X. in The calculation method is as follows:

[0023] X in =σ(BN(Conv) 7×7 (X5)))

[0024] Where Conv7×7(·) represents a 7×7 convolution with a stride of 1, BN(·) is a batch normalization layer, and σ(·) is the ReLU activation function;

[0025] S3.1.2: Transfer feature X in Intermediate features were obtained after PConv module and group normalization processing. The calculation method is as follows:

[0026]

[0027] Where GN(·) is group normalization and PConv(·) is partial convolution;

[0028] S3.1.3: Features The global long-term dependency X is obtained after processing by the residual module. MLP The calculation method is as follows:

[0029]

[0030] Wherein, CMLP(·) is the channel MLP.

[0031] Furthermore, in step S3.2 above, the aggregation of local region features within the deepest feature layer is achieved, i.e., X LVC The calculation method is as follows:

[0032] S3.2.1: X from the Stem module in The features are encoded through a set of convolutional layers to obtain the information e of the Kth codeword in the entire image. k The calculation method is as follows:

[0033]

[0034] in, Represents the i-th pixel, b k For the k-th learnable visual codeword, s k It is the kth scaling factor. It is information about the position of each pixel relative to the codeword;

[0035] S3.2.2: e k After processing through a normalization layer, activation layer, and mean layer, the information e of the entire image for K codewords is obtained, calculated as follows:

[0036]

[0037] Where φ represents normalization, activation, and averaging;

[0038] S3.2.3: Input the information e into a fully connected layer and a 1×1 convolution to highlight the key feature Z. The calculation method is as follows:

[0039]

[0040] in, It is a channel-wise element-wise multiplication, and δ(·) is the sigmoid function;

[0041] S3.2.4: Obtain X LVC The calculation method is as follows:

[0042]

[0043] in, It is an element-by-element addition of channels.

[0044] Furthermore, the construction of the TAL-AAT detection head in step S4 above specifically includes:

[0045] S4.1: Obtain the highest classification score and the most accurate bounding box by learning TAL through task alignment;

[0046] S4.2: Anchor frame-assisted training is introduced, which fully combines the advantages of anchor frame-based and anchor frame-free training modes.

[0047] Furthermore, in step S4.1 above, the highest classification score and the most accurate bounding box are obtained through task alignment learning TAL, calculated as follows:

[0048] S4.1.1: The task alignment t is measured using a higher-order combination of classification scores and IoU, calculated as follows:

[0049] t=sα ×u β

[0050] Where s and u represent the classification score and IoU value, respectively, and α and β are used to control the influence on the anchor alignment metric;

[0051] S4.1.2: Replace the binary label of the positive anchor point with a normalized t, calculated as follows:

[0052]

[0053] Where i represents the i-th anchor point among the positive anchor points corresponding to a target. t represents the normalized value;

[0054] S4.1.3: Focus loss is used for classification. The final loss function for the classification task is defined as follows:

[0055]

[0056] Where j represents the j-th anchor in the negative anchor points, and γ is the focusing parameter;

[0057] S4.1.4: The loss function for the regression task is defined as follows:

[0058]

[0059] Among them, b and Represents the predicted bounding box and the corresponding ground truth box;

[0060] S4.1.5: The final loss function is defined as follows:

[0061] L loss =L cls +L reg .

[0062] Furthermore, step S4.2 above introduces anchor-frame-assisted training, specifically including:

[0063] During the training phase, the anchor-assisted branch and the unanchored branch perform loss calculations and gradient propagation simultaneously. During inference, the auxiliary head is removed.

[0064] The beneficial effects of this invention are as follows:

[0065] 1. This invention can accurately detect swimmers in various scenarios, such as when swimmers are close by or when drowning victims are mostly in the water, without any missed detections.

[0066] 2. In a swimming pool environment with swimming rings and toys, it can effectively distinguish swimming rings and toys from swimmers, resulting in better detection accuracy and robustness.

[0067] 3. When the identification results detect drowning behavior in the area, the system will promptly issue an alarm to notify the safety officer, which improves the speed and accuracy of the identification while making up for the shortcomings of relying solely on the safety officer for monitoring.

[0068] 4. The hardware configuration is relatively simple, the cost is low, the operation is convenient, and the effect is obvious. It is of great significance to reduce the occurrence of drowning incidents and protect the lives of swimmers. Attached Figure Description

[0069] Figure 1 Flowchart of the detection method of this invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0071] like Figure 1 As shown, this invention is a swimming pool drowning detection method based on RTMDet, comprising the following steps:

[0072] S1) Use video capture equipment to capture pool video frames, extract images from the video frames at fixed intervals, label the images to obtain a pool image dataset, and divide the pool image dataset into training set images and validation set images. The training set images are used to train the network model, and the validation set images are used to evaluate the model performance and adjust the model parameters during the training process.

[0073] As a preferred embodiment of the present invention, when constructing a swimming pool drowning detection model, it is necessary to use a camera to capture swimming videos in real-world scenarios and collect relevant videos, then extract an image every 10 frames, and then use the Labelme annotation tool to annotate the images, and then convert the annotation files into the COCO dataset format.

[0074] S2) Construct a heterogeneous feature extraction backbone network to extract multi-scale features of different sizes and resolutions from the training set images. The multi-scale features include features of different sizes and resolutions extracted by different depths of the backbone network.

[0075] In a preferred embodiment of the present invention, the construction of the heterogeneous feature extraction backbone network in step S2 includes the following steps:

[0076] S2.1: By setting the kernel size K in the backbone network to 3, 5, 7, and 9 in sequence, interference information other than the effective information of small objects can be reduced;

[0077] S2.2: Reduce model computation and speed up inference by using Pconv;

[0078] The Pconv method applies conventional convolution to extract spatial features for some input channels, while leaving the remaining channels unchanged.

[0079] S2.3: Improve the model's attention to key features by using the SimAM attention module.

[0080] SimAM is a similarity-adaptive module. SimAM draws on the spatial inhibition theory in neuroscience, which states that neurons containing the most information typically have different firing patterns than surrounding neurons, and neurons around active neurons are inhibited. Therefore, neurons exhibiting significant spatial inhibition effects in computer vision tasks are assigned higher weights.

[0081] In a preferred embodiment of the present invention, the original image is first processed through three 3×3 convolutions, and then through four heterogeneous Sim-CSP modules for feature extraction. The kernel sizes of the four heterogeneous Sim-CSP modules are 3, 5, 7, and 9, respectively. Each Sim-CSP module includes a convolutional layer, a batch normalization layer, an activation layer, a module including a residual connection, and a SimAM attention module. This heterogeneous feature extraction network extracts 80×80, 40×40, and 20×20 feature maps, respectively, and then these feature maps of different scales are input into the CFP-FPN module for multi-scale feature fusion.

[0082] S3) Construct a CFP-FPN multi-scale feature pyramid, and input the multi-scale features extracted in S2) into the CFP-FPN multi-scale feature pyramid for fusion, so as to aggregate features of different resolutions to achieve effective detection of targets at different scales.

[0083] In a preferred embodiment of the present invention, the construction of the CFP-FPN module in step S3 specifically includes:

[0084] S3.1: Capturing Global Long-Term Dependencies X of Deepest Features Using a Lightweight MLP MLP (i.e., global information);

[0085] S3.2: The aggregation of local region features (i.e., local information) within the deepest feature layer is achieved through learnable visual center LVC, i.e., X.LVC ;

[0086] S3.3: X MLP and X LVC The pieces are stitched together along the channel dimension.

[0087] In a preferred embodiment of the present invention, step S3.1 captures the global long-term dependency X of the deepest feature. MLP The calculation method is as follows:

[0088] S3.1.1: The deepest feature X5 extracted from the backbone network is processed through a Stem block containing convolutional layers, normalization layers, and ReLU activation functions to obtain feature X. in The calculation method is as follows:

[0089] X in =σ(BN(Conv) 7×7 (X5)))

[0090] Where Conv7×7(·) represents a 7×7 convolution with a stride of 1, BN(·) is a batch normalization layer, and σ(·) is the ReLU activation function;

[0091] S3.1.2: Transfer feature X in Intermediate features were obtained after PConv module and group normalization processing. The calculation method is as follows:

[0092]

[0093] Where GN(·) is group normalization and PConv(·) is partial convolution;

[0094] S3.1.3: Features The global long-term dependency X is obtained after processing by the residual module. MLP The calculation method is as follows:

[0095]

[0096] Wherein, CMLP(·) is the channel MLP.

[0097] In a preferred embodiment of the present invention, step S3.2 involves the aggregation of local region features within the deepest feature layer, i.e., X. LVC The calculation method is as follows:

[0098] S3.2.1: X from the Stem module in The features are encoded through a set of convolutional layers to obtain the information e of the Kth codeword in the entire image. k The calculation method is as follows:

[0099]

[0100] in, Represents the i-th pixel, b k For the k-th learnable visual codeword, s k It is the kth scaling factor. It is information about the position of each pixel relative to the codeword;

[0101] S3.2.2: e k After processing through a normalization layer, activation layer, and mean layer, the information e of the entire image for K codewords is obtained, calculated as follows:

[0102]

[0103] Where φ represents normalization, activation, and averaging;

[0104] S3.2.3: Input the information e into a fully connected layer and a 1×1 convolution to highlight the key feature Z. The calculation method is as follows:

[0105]

[0106] in, It is a channel-wise element-wise multiplication, and δ(·) is the sigmoid function;

[0107] S3.2.4: Obtain X LVC The calculation method is as follows:

[0108]

[0109] in, It is an element-by-element addition of channels.

[0110] S4) Construct the TAL-AAT detection head, input the aggregated features from S3) into the TAL-AAT detection head, and combine the classification score, the most accurate bounding box, and the gradient of the model based on anchor boxes and no anchor boxes to train the network, thereby improving the accuracy of model classification and localization.

[0111] In a preferred embodiment of the present invention, the construction of the TAL-AAT detection head in step S4 specifically includes:

[0112] S4.1: Obtain the highest classification score and the most accurate bounding box by learning TAL through task alignment;

[0113] In a preferred embodiment of the present invention, in step S4.1, the highest classification score and the most accurate bounding box are obtained through task alignment learning TAL. The calculation method is as follows:

[0114] S4.1.1: The task alignment t is measured using a higher-order combination of classification scores and IoU, calculated as follows:

[0115] t=sα ×u β

[0116] Where s and u represent the classification score and IoU value, respectively, and α and β are used to control the influence on the anchor alignment metric;

[0117] S4.1.2: Replace the binary label of the positive anchor point with a normalized t, calculated as follows:

[0118]

[0119] Where i represents the i-th anchor point among the positive anchor points corresponding to a target. t represents the normalized value;

[0120] S4.1.3: Focus loss is used for classification. The final loss function for the classification task is defined as follows:

[0121]

[0122] Where j represents the j-th anchor in the negative anchor points, and Y is the focus parameter;

[0123] S4.1.4: The loss function for the regression task is defined as follows:

[0124]

[0125] Among them, b and Represents the predicted bounding box and the corresponding ground truth box;

[0126] S4.1.5: The final loss function is defined as follows:

[0127] L loss =L cls +L reg .

[0128] S4.2: Anchor frame-assisted training is introduced, which fully combines the advantages of anchor frame-based and anchor frame-free training modes.

[0129] In a preferred embodiment of the present invention, step S4.2 introduces anchor frame-assisted training, specifically including:

[0130] During the training phase, the anchor-assisted branch and the unanchored branch perform loss calculations and gradient propagation simultaneously. During inference, the auxiliary head is removed.

[0131] S5) Iteratively train the model using the training set images and the validation set images until the loss function converges, and save the trained model. The loss function in step S5) is the same as the loss function in step S4.1.

[0132] This invention uses the AdamW optimizer and the flat-cosine method during the training process to adopt a fixed learning rate in the first half of the training cycle, and uses the cosine annealing method to adjust the learning rate in the second half.

[0133] S6) Set up multiple cameras to monitor the swimming pool in real time and transmit the video to the drowning detection and recognition system. The system uses the swimming pool drowning detection model to output the detection results. When the system detects that drowning has occurred, it will issue an alarm and save the detected video.

[0134] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A swimming pool drowning detection method based on RTMDet, characterized in that, Includes the following steps: S1) Use video capture equipment to capture pool video frames, extract images from the video frames at fixed frame intervals, label the images to obtain a pool image dataset, and divide the pool image dataset into training set images and validation set images. The training set images are used to train the network model, and the validation set images are used to evaluate the model performance and adjust the model parameters during the training process. S2) Construct a heterogeneous feature extraction backbone network to extract multi-scale features of different sizes and resolutions from the training set images; S3) Construct a CFP-FPN multi-scale feature pyramid, and input the multi-scale features extracted in S2) into the CFP-FPN multi-scale feature pyramid for fusion, thereby aggregating features of different resolutions to achieve effective detection of targets at different scales; S4) Construct the TAL-AAT detection head, input the aggregated features from S3) into the TAL-AAT detection head, and combine the classification score, the most accurate bounding box, and the gradient of the model based on anchor boxes and no anchor boxes to train the network, thereby improving the accuracy of model classification and localization. S5) Use the training set images and validation set images to iteratively train the model until the loss function converges, and save the trained model. S6) Set up multiple cameras to monitor the swimming pool in real time and transmit the video to the drowning detection and recognition system. The system uses the swimming pool drowning detection model to output the detection results. When the system detects that drowning has occurred, it will issue an alarm and save the detected video. The construction of the heterogeneous feature extraction backbone network in S2 includes the following steps: S2.1: By setting the kernel size K in the backbone network to 3, 5, 7, and 9 in sequence, interference information other than the effective information of small objects can be reduced; S2.2: Reduce model computation and speed up inference by using Pconv; S2.3: Improve the model's attention to key features through the SimAM attention module; The construction of the CFP-FPN module in S3 specifically includes: S3.1: Capturing global long-term dependencies of deepest features using a lightweight MLP ; S3.2: The aggregation of local region features within the deepest feature layer is achieved through learnable visual center LVC (Learning Visual Center Convergence). ; S3.3: Will and Stitch along the channel dimension; The construction of the TAL-AAT detection head in S4 specifically includes: S4.1: Obtain the highest classification score and the most accurate bounding box by learning TAL through task alignment; S4.2: Anchor frame-assisted training is introduced, which fully combines the advantages of anchor frame-based and anchor frame-free training modes.

2. The swimming pool drowning detection method based on RTMDet according to claim 1, characterized in that, S3.1 captures the global long-term dependencies of the deepest features. The calculation method is as follows: S3.1.1: Extract the deepest features from the backbone network Features are obtained by smoothing the features using a Stem block containing convolutional layers, normalized layers, and ReLU activation functions. The calculation method is as follows: Where Conv7×7(·) represents a 7×7 convolution with a stride of 1, and BN(·) is a batch normalization layer. (·) is the ReLU activation function; S3.1.2: Features Intermediate features were obtained after PConv module and group normalization processing. The calculation method is as follows: , where GN(·) is group normalization and PConv(·) is partial convolution; S3.1.3: Intermediate features The global long-term dependencies are obtained after processing by the residual module. The calculation method is as follows: , where CMLP(·) is the channel MLP.

3. The swimming pool drowning detection method based on RTMDet according to claim 1, characterized in that, In S3.2, the aggregation of local region features within the deepest feature layer is achieved, i.e. The calculation method is as follows: S3.2.1: From the Stem module The features are encoded through a set of convolutional layers to obtain the information of the Kth codeword in the entire image. The calculation method is as follows: ,in, Represents the i-th pixel. For the k-th learnable visual codeword, It is the kth scaling factor. It is information about the position of each pixel relative to the codeword; S3.2.2: will After processing through a normalization layer, activation layer, and mean layer, the information of the entire image for K codewords is obtained. The calculation method is as follows: ,in, Represents normalization, activation, and averaging; S3.2.3: Send information Input to fully connected layer and 1 Convolution with 1 to highlight key features The calculation method is as follows: ,in, It involves element-wise multiplication of channels. (·) is the sigmoid function; S3.2.4: Obtain The calculation method is as follows: ,in, It is an element-by-element addition of channels.

4. The swimming pool drowning detection method based on RTMDet according to claim 1, characterized in that, In step S4.1, the highest classification score and the most accurate bounding box are obtained through task alignment learning TAL. The calculation method is as follows: S4.1.1: Use a higher-order combination of classification scores and IoU to measure task alignment. The calculation method is as follows: Where s and u represent the classification score and IoU value, respectively. and Used to control the impact on anchor alignment metrics; S4.1.2: Replace the binary label of the positive anchor point with a normalized t, calculated as follows: Where i represents the i-th anchor point among the positive anchor points corresponding to a target. t represents the normalized value; S4.1.3: Focus loss is used for classification. The final loss function for the classification task is defined as follows: Where j represents the j-th anchor in the negative anchor points, and γ is the focusing parameter; S4.1.4: The loss function for the regression task is defined as follows: , where b and Represents the predicted bounding box and the corresponding ground truth box; S4.1.5: The final loss function is defined as follows: 。 5. The swimming pool drowning detection method based on RTMDet according to claim 1, characterized in that, S4.2 introduces anchor-frame-assisted training, specifically including: During the training phase, the anchor-assisted branch and the unanchored branch perform loss calculations and gradient propagation simultaneously. During inference, the auxiliary head is removed.

Images