Swimming pool drowning detection method based on RTMDet

Abstract

The invention discloses a swimming pool drowning detection method based on RTMDet, and the method comprises the steps: building a heterogeneous feature extraction backbone network and a CFP-FPN module, carrying out the task alignment learning, carrying out the auxiliary training through an anchor frame, combining the features based on the anchor frame and without the anchor frame, improving the effective receptive field of a model while not remarkably increasing the complexity of the model, and achieving the detection of drowning of a swimming pool. And the detection precision of swimmers with different scales in a complex scene is improved. The CFP-FPN module provided by the invention can fully fuse interlayer and intra-layer features, the detection accuracy of the model for different scale targets is improved, and the provided TAL-AAT detection head fully combines the classification score and the most accurate bounding box and improves the accuracy of model classification and positioning based on the advantages of an anchor frame and without an anchor frame.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a swimming pool drowning detection method based on RTMDet. Background Art

[0002] Swimming can improve children's physical fitness, coordination ability, intelligence level, and promote neurological development. Therefore, more and more parents encourage their children to participate in swimming, but swimming has hidden dangers that cannot be ignored. According to statistics, about 45% of drowning deaths worldwide are children. In China, drowning has become one of the three major causes of accidental injury and death among children. With the popularity of swimming, related safety issues have attracted widespread attention. Although children are protected by swimming rings when swimming, accidents are still mainly caused by the negligence of safety lifeguards. Safety guards are easily affected by factors such as emotional fluctuations and fatigue, which affects the effect of monitoring. In recent years, many cases of child drowning have occurred in my country. These incidents are devastating to families and also show that child drowning is still a serious public safety issue. It can be seen that the method of simply using safety lifeguards to monitor swimming has certain disadvantages and cannot ensure the safety of swimmers. It is of great practical significance to study and apply computer vision methods to assist safety lifeguards in monitoring swimming pools to effectively reduce the probability of accidents.

[0003] With the successful application of target detection in many fields, drowning detection based on target detection algorithms has become the mainstream. At present, target detection algorithms based on deep learning can be divided into two types: two-stage algorithms and single-stage algorithms. The two-stage algorithm first generates a candidate box from the image, and then uses the learning module to extract the regional features of the candidate box and complete the final classification and regression. Representative algorithms include the R-CNN series. This type of algorithm has high detection accuracy but slow detection speed, which cannot meet the real-time detection scenario. The single-stage algorithm directly regresses and classifies the target. Representative algorithms include SSD, YOLO series and RTMDet. This type of algorithm has a faster detection speed and higher detection accuracy, so the single-stage target detection algorithm has been widely used.

[0004] In order to detect drowning behavior, researchers at home and abroad have proposed many methods, including underwater human body detection method based on background difference method, swimming pool drowning behavior detection method based on Mask R-CNN, and swimming pool drowning detection method based on YOLOv4. However, these methods have their own defects: 1. The background modeling of the background difference method will be affected by many dynamic factors in the scene, which will affect the subsequent target extraction and target tracking, making the method have low detection accuracy in complex environments. 2. The detection accuracy of the detection method based on Mask R-CNN has been improved, but the detection speed of this method is slow. 3. The underwater detection method based on YOLOv4 is easily blocked by swimmers and the maintenance cost of the underwater camera required by this method is high. 4. Most swimming pool drowning detection methods have good detection accuracy in simple scenes, but the detection accuracy is low in complex scenes such as mixed swimming rings and toys in the swimming pool, reflective interference on the water surface, and diverse swimmer shapes, which is prone to misidentification of swimmers. Summary of the invention

[0005] The purpose of the present invention is to provide an efficient and accurate detection system for drowning accidents in swimming pools, so as to improve the detection accuracy of drowning accidents in complex scenes of swimming pools.

[0006] To achieve the above-mentioned object of the invention, the present invention proposes a swimming pool drowning detection method based on RTMDet, comprising the following steps:

[0007] S1) using a video acquisition device to collect swimming pool video frames, extracting images from the video frames at fixed frame intervals, annotating the images, obtaining a swimming pool image dataset, and dividing the swimming pool image dataset into a training set image and a validation set image, wherein the training set image is used to train the network model, and the validation set image is used to evaluate the model effect during the training process and adjust the model parameters;

[0008] S2) constructing a heterogeneous feature extraction backbone network to extract multi-scale features of different sizes and resolutions in the training set images;

[0009] S3) constructing a CFP-FPN multi-scale feature pyramid, and inputting the multi-scale features extracted in S2) into the CFP-FPN multi-scale feature pyramid for fusion, so as to aggregate features of different resolutions to achieve effective detection of objects of different scales;

[0010] S4) constructing a TAL-AAT detection head, inputting the aggregated features in S3) into the TAL-AAT detection head, combining the classification score, the most accurate bounding box, the anchor box and the anchor-free box to calculate the gradient of the model to train the network, so as to improve the accuracy of model classification and positioning;

[0011] S5) iteratively training the model using the training set images and the validation set images until the loss function converges, and saving the trained model;

[0012] S6) multiple cameras are set up to monitor the swimming pool in real time and transmit the video to the drowning detection and recognition system. The system uses the swimming pool drowning detection model to output the detection results. When the system detects that drowning has occurred, it will issue an alarm and save the detected video.

[0013] Furthermore, the construction of the heterogeneous feature extraction backbone network in the above step S2 includes the following steps:

[0014] S2.1: The size of the convolution kernel K is set to 3, 5, 7, and 9 in the backbone network to reduce the interference information other than the effective information of small objects;

[0015] S2.2: Use Pconv to reduce the amount of model calculation and speed up the model reasoning;

[0016] S2.3: Improve the model’s attention to key features through the SimAM attention module.

[0017] Furthermore, the construction of the CFP-FPN module in the above step S3 specifically includes:

[0018] S3.1: Capturing the global long-term dependencies of the deepest features through a lightweight MLP MLP ;

[0019] S3.2: Aggregation of local region features within the deepest feature layer is achieved through learnable visual center LVC, i.e. X LVC ;

[0020] S3.3: X MLP and X LVc Stitching is done along the channel dimension.

[0021] Furthermore, the global long-term dependency X of the deepest features captured in the above step S3.1 MLP , the calculation method is as follows:

[0022] S3.1.1: The deepest feature X5 extracted by the backbone network is smoothed by the Stem block containing the convolution layer and the normalization layer ReLU activation function to obtain the feature X in , the calculation method is as follows:

[0023] X in =σ(BN(Conv 7×7 (X5)))

[0024] Among them, Conv7×7(·) represents a 7×7 convolution with stride 1, BN(·) is a batch normalization layer, and σ(·) is a ReLU activation function;

[0025] S3.1.2: Transform feature X in After PConv module and group normalization processing, the intermediate features are obtained The calculation method is as follows:

[0026]

[0027] Among them, GN(·) is group normalization, PConv(·) is partial convolution;

[0028] S3.1.3: Characterize After being processed by the residual module, the global long-term dependency X is obtained MLP , the calculation method is as follows:

[0029]

[0030] Among them, CMLP(·) is the channel MLP.

[0031] Furthermore, the above step S3.2 realizes the aggregation of local area features in the deepest feature layer, that is, X LVC , the calculation method is as follows:

[0032] S3.2.1: X from the Stem module in The features are encoded through a set of convolutional layers to obtain the information e of the entire image for the Kth codeword k , the calculation method is as follows:

[0033]

[0034] in, represents the i-th pixel, b k is the kth learnable visual codeword, s k is the kth scaling factor, It is the information of each pixel’s position relative to the codeword;

[0035] S3.2.2: e k After processing by the normalization layer, activation layer and mean layer, the information e of the entire image for K codewords is obtained. The calculation method is as follows:

[0036]

[0037] Among them, φ represents normalization, activation and averaging;

[0038] S3.2.3: Input the information e into the fully connected layer and 1×1 convolution to highlight the key feature Z. The calculation method is as follows:

[0039]

[0040] in, is the channel element-by-element multiplication, δ(·) is the sigmoid function;

[0041] S3.2.4: Get X LVC , the calculation method is as follows:

[0042]

[0043] in, It is the channel element-by-element addition.

[0044] Furthermore, the construction of the TAL-AAT detection head in the above step S4 specifically includes:

[0045] S4.1: Learn TAL by task alignment to obtain the highest classification score and the most accurate bounding box;

[0046] S4.2: Anchor-assisted training is introduced to fully combine the advantages of anchor-based and anchor-free training modes.

[0047] Furthermore, in the above step S4.1, the highest classification score and the most accurate bounding box are obtained by learning TAL through task alignment, and the calculation method is as follows:

[0048] S4.1.1: Use a high-order combination of the classification score and IoU to measure the task alignment t, calculated as follows:

[0049] t=sα ×u β

[0050] Among them, s and u represent the classification score and IoU value respectively, and α and β are used to control the impact on the anchor alignment metric;

[0051] S4.1.2: Use a normalized t to replace the binary label of the positive anchor point, calculated as follows:

[0052]

[0053] Among them, i represents the i-th anchor point among the positive anchor points corresponding to an object, represents the normalized t;

[0054] S4.1.3: Focal loss is used for classification. The final loss function of the classification task is defined as follows:

[0055]

[0056] Where j represents the jth anchor in the negative anchor point, and γ is the focusing parameter;

[0057] S4.1.4: The loss function for the regression task is defined as follows:

[0058]

[0059] Among them, b and Represents the predicted bounding box and the corresponding ground-truth box;

[0060] S4.1.5: The final loss function is defined as follows:

[0061] L loss =L cls +L reg .

[0062] Furthermore, the above step S4.2 introduces anchor frame auxiliary training, which specifically includes:

[0063] During the training phase, the anchor-assisted branch and the anchor-free branch perform loss calculations and gradient propagation respectively, and the auxiliary head is removed during inference.

[0064] The beneficial effects of the present invention are as follows:

[0065] 1. The present invention can accurately detect swimmers in multiple scenarios, such as when the swimmers are relatively close and when most of the drowning people are in the water, without any missed detection.

[0066] 2. In an environment where there are swimming rings and toys in the swimming pool, the swimming rings and toys can be well distinguished from swimmers, with better detection accuracy and robustness.

[0067] 3. When the recognition results show that drowning has occurred in the area, the system will promptly issue an alarm to notify the safety officer, which improves the speed and accuracy of the identification while making up for the drawbacks of relying solely on safety officers for monitoring.

[0068] 4. The hardware configuration is relatively simple, the cost is low, the operation is convenient, and the effect is obvious. It is of great significance to reduce the occurrence of drowning incidents and ensure the safety of swimmers. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 Flow chart of the detection method of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0071] like Figure 1 As shown, the present invention is a swimming pool drowning detection method based on RTMDet, comprising the following steps:

[0072] S1) using a video acquisition device to collect swimming pool video frames, extracting images from the video frames at fixed frame intervals, annotating the images, obtaining a swimming pool image dataset, and dividing the swimming pool image dataset into a training set image and a validation set image. The training set images are used to train the network model, and the validation set images are used to evaluate the model effect and adjust the model parameters during the training process.

[0073] As a preferred embodiment of the present invention, when constructing a swimming pool drowning detection model, it is necessary to use a camera to shoot swimming videos in real scenes and collect related videos, then extract a picture every 10 frames, and then use the Labelme annotation tool to annotate them, and then convert the annotation file into the format of the COCO dataset.

[0074] S2) constructing a heterogeneous feature extraction backbone network to extract multi-scale features of different sizes and resolutions in the training set images. The multi-scale features include: features of different sizes and resolutions extracted at different depths of the backbone network.

[0075] As a preferred embodiment of the present invention, constructing a heterogeneous feature extraction backbone network in step S2 includes the following steps:

[0076] S2.1: The size of the convolution kernel K is set to 3, 5, 7, and 9 in the backbone network to reduce the interference information other than the effective information of small objects;

[0077] S2.2: Use Pconv to reduce the amount of model calculation and speed up the model reasoning;

[0078] The Pconv applies conventional convolution to some input channels for spatial feature extraction, while keeping the remaining channels unchanged.

[0079] S2.3: Improve the model’s attention to key features through the SimAM attention module.

[0080] The SimAM is a similarity adaptive module. It draws on the spatial inhibition theory in neuroscience, that is, neurons containing the most information usually have different discharge patterns from surrounding neurons and neurons around active neurons are inhibited. Therefore, neurons that show obvious spatial inhibition effects in computer vision tasks are given higher weights.

[0081] As a preferred embodiment of the present invention, the present invention first passes the original image through three 3×3 convolutions, and then passes it through four heterogeneous Sim-CSP modules for feature extraction, wherein the convolution kernel sizes of the four heterogeneous Sim-CSP modules are 3, 5, 7, and 9 respectively. Each Sim-CSP module includes a convolution layer, a batch normalization layer, an activation layer, a residual connection module, and a SimAM attention module. The heterogeneous feature extraction network is used to extract feature maps of 80×80, 40×40, and 20×20 respectively, and then the feature maps of different scales are input into the CFP-FPN module for multi-scale feature fusion.

[0082] S3) constructs a CFP-FPN multi-scale feature pyramid, and inputs the multi-scale features extracted in S2) into the CFP-FPN multi-scale feature pyramid for fusion, so as to aggregate features of different resolutions to achieve effective detection of targets of different scales.

[0083] As a preferred embodiment of the present invention, constructing the CFP-FPN module in step S3 specifically includes:

[0084] S3.1: Capturing the global long-term dependencies of the deepest features through a lightweight MLP MLP (i.e. global information);

[0085] S3.2: Aggregation of local features (i.e., local information) within the deepest feature layer is achieved through the learnable visual center LVC, i.e., X LVC ;

[0086] S3.3: X MLP and X LVC Stitching is done along the channel dimension.

[0087] As a preferred embodiment of the present invention, the global long-term dependency X of the deepest features captured in step S3.1 MLP , the calculation method is as follows:

[0088] S3.1.1: The deepest feature X5 extracted by the backbone network is smoothed by the Stem block containing the convolution layer and the normalization layer ReLU activation function to obtain the feature X in , the calculation method is as follows:

[0089] X in =σ(BN(Conv 7×7 (X5)))

[0090] Among them, Conv7×7(·) represents a 7×7 convolution with stride 1, BN(·) is a batch normalization layer, and σ(·) is a ReLU activation function;

[0091] S3.1.2: Transform feature Xin After PConv module and group normalization processing, the intermediate features are obtained The calculation method is as follows:

[0092]

[0093] Among them, GN(·) is group normalization, PConv(·) is partial convolution;

[0094] S3.1.3: Characterize After being processed by the residual module, the global long-term dependency X is obtained MLP , the calculation method is as follows:

[0095]

[0096] Among them, CMLP(·) is the channel MLP.

[0097] As a preferred embodiment of the present invention, step S3.2 realizes the aggregation of local area features in the deepest feature layer, that is, X LVC , the calculation method is as follows:

[0098] S3.2.1: X from the Stem module in The features are encoded through a set of convolutional layers to obtain the information e of the entire image for the Kth codeword k , the calculation method is as follows:

[0099]

[0100] in, represents the i-th pixel, b k is the kth learnable visual codeword, s k is the kth scaling factor, It is the information of each pixel’s position relative to the codeword;

[0101] S3.2.2: e k After processing by the normalization layer, activation layer and mean layer, the information e of the entire image for K codewords is obtained. The calculation method is as follows:

[0102]

[0103] Among them, φ represents normalization, activation and averaging;

[0104] S3.2.3: Input the information e into the fully connected layer and 1×1 convolution to highlight the key feature Z. The calculation method is as follows:

[0105]

[0106] in, is the channel element-by-element multiplication, δ(·) is the sigmoid function;

[0107] S3.2.4: Get X LVC , the calculation method is as follows:

[0108]

[0109] in, It is the channel element-by-element addition.

[0110] S4) constructs a TAL-AAT detection head, inputs the aggregated features in S3) into the TAL-AAT detection head, and combines the classification score, the most accurate bounding box, the anchor box and the anchor-free box to calculate the gradient of the model to train the network, so as to improve the accuracy of model classification and positioning.

[0111] As a preferred embodiment of the present invention, constructing the TAL-AAT detection head in step S4 specifically includes:

[0112] S4.1: Learn TAL by task alignment to obtain the highest classification score and the most accurate bounding box;

[0113] As a preferred embodiment of the present invention, in step S4.1, the highest classification score and the most accurate bounding box are obtained by task alignment learning TAL, and the calculation method is as follows:

[0114] S4.1.1: Use a high-order combination of the classification score and IoU to measure the task alignment t, calculated as follows:

[0115] t=sα ×u β

[0116] Among them, s and u represent the classification score and IoU value respectively, and α and β are used to control the impact on the anchor alignment metric;

[0117] S4.1.2: Use a normalized t to replace the binary label of the positive anchor point, calculated as follows:

[0118]

[0119] Among them, i represents the i-th anchor point among the positive anchor points corresponding to an object, represents the normalized t;

[0120] S4.1.3: Focal loss is used for classification. The final loss function of the classification task is defined as follows:

[0121]

[0122] Where j represents the jth anchor in the negative anchor point, and Y is the focus parameter;

[0123] S4.1.4: The loss function for the regression task is defined as follows:

[0124]

[0125] Among them, b and Represents the predicted bounding box and the corresponding ground-truth box;

[0126] S4.1.5: The final loss function is defined as follows:

[0127] L loss =L cls +L reg .

[0128] S4.2: Anchor-assisted training is introduced to fully combine the advantages of anchor-based and anchor-free training modes.

[0129] As a preferred embodiment of the present invention, anchor frame auxiliary training is introduced in step S4.2, which specifically includes:

[0130] During the training phase, the anchor-assisted branch and the anchor-free branch perform loss calculations and gradient propagation respectively, and the auxiliary head is removed during inference.

[0131] S5) Iteratively train the model using the training set images and the validation set images until the loss function converges, and save the trained model. The loss function in step S5) is the same as the loss function in step S4.1.

[0132] The present invention uses the AdamW optimizer and the flat-cosine method in the training process to adopt a fixed learning rate in the first half of the training cycle, and uses the cosine annealing method to adjust the learning rate in the second half.

[0133] S6) multiple cameras are set up to monitor the swimming pool in real time and transmit the video to the drowning detection and recognition system. The system uses the swimming pool drowning detection model to output the detection results. When the system detects that drowning has occurred, it will issue an alarm and save the detected video.

[0134] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A swimming pool drowning detection method based on RTMDet, characterized in that: The steps include: S1) using a video acquisition device to collect swimming pool video frames, extracting images from the video frames at fixed frame intervals, annotating the images, obtaining a swimming pool image dataset, and dividing the swimming pool image dataset into a training set image and a validation set image, wherein the training set image is used to train the network model, and the validation set image is used to evaluate the model effect during the training process and adjust the model parameters; S2) constructing a heterogeneous feature extraction backbone network to extract multi-scale features of different sizes and resolutions in the training set images; S3) constructing a CFP-FPN multi-scale feature pyramid, and inputting the multi-scale features extracted in S2) into the CFP-FPN multi-scale feature pyramid for fusion, so as to aggregate features of different resolutions to achieve effective detection of objects of different scales; S4) constructing a TAL-AAT detection head, inputting the aggregated features in S3) into the TAL-AAT detection head, combining the classification score, the most accurate bounding box, the anchor box and the anchor-free box to calculate the gradient of the model to train the network, so as to improve the accuracy of model classification and positioning; S5) iteratively training the model using the training set images and the validation set images until the loss function converges, and saving the trained model; S6) multiple cameras are set up to monitor the swimming pool in real time and transmit the video to the drowning detection and recognition system. The system uses the swimming pool drowning detection model to output the detection results. When the system detects that drowning has occurred, it will issue an alarm and save the detected video.

2. A swimming pool drowning detection method based on RTMDet according to claim 1, characterized in that: The construction of the heterogeneous feature extraction backbone network in S2 includes the following steps S2.1: The size of the convolution kernel K is set to 3, 5, 7, and 9 in the backbone network to reduce the interference information other than the effective information of small objects; S2.2: Use Pconv to reduce the amount of model calculation and speed up the model reasoning; S2.3: Improve the model’s attention to key features through the SimAM attention module.

3. A swimming pool drowning detection method based on RTMDet according to claim 1, characterized in that: The construction of the CFP-FPN module in S3 specifically includes: S3.1: Capturing the global long-term dependencies of the deepest features through a lightweight MLP MLP ; S3.2: Aggregation of local region features within the deepest feature layer is achieved through learnable visual center LVC, i.e. X LVC ; S3.3: X MLP and X LVC Stitching is done along the channel dimension.

4. A swimming pool drowning detection method based on RTMDet according to claim 3, characterized in that: The global long-term dependency X of the deepest features is captured in S3.1 MLP , the calculation method is as follows: S3.1.1: The deepest feature X5 extracted by the backbone network is smoothed by the Stem block containing the convolution layer and the normalization layer ReLU activation function to obtain the feature X in , the calculation method is as follows: X in =σ(BN(Conv 7×7 (X5))) Among them, Conv7×7(·) represents a 7×7 convolution with stride 1, BN(·) is a batch normalization layer, and σ(·) is a ReLU activation function; S3.1.2: Feature X in After PConv module and group normalization processing, the intermediate features are obtained The calculation method is as follows: Among them, GN(·) is group normalization, PConv(·) is partial convolution; S3.1.3: Intermediate features After being processed by the residual module, the global long-term dependency X is obtained MLP , the calculation method is as follows: Among them, CMLP(·) is the channel MLP.

5. A swimming pool drowning detection method based on RTMDet according to claim 3, characterized in that: The S3.2 realizes the aggregation of local area features in the deepest feature layer, that is, X LVC , the calculation method is as follows: S3.2.1: X from the Stem module in The features are encoded through a set of convolutional layers to obtain the information e of the entire image for the Kth codeword k , the calculation method is as follows: in, represents the i-th pixel, b k is the kth learnable visual codeword, s k is the kth scaling factor, -b k It is the information of each pixel’s position relative to the codeword; S3.2.2: e k After processing by the normalization layer, activation layer and mean layer, the information e of the entire image for K codewords is obtained. The calculation method is as follows: Among them, φ represents normalization, activation and averaging; S3.2.3: Input the information w into the fully connected layer and 1×1 convolution to highlight the key feature Z. The calculation method is as follows: in, is the channel element-by-element multiplication, δ(·) is the sigmoid function; S3.2.4: Get X LVC , the calculation method is as follows: in, It is the channel element-by-element addition.

6. A swimming pool drowning detection method based on RTMDet according to claim 1, characterized in that: The construction of the TAL-AAT detection head in S4 specifically includes: S4.1: Learn TAL by task alignment to obtain the highest classification score and the most accurate bounding box; S4.2: Anchor-assisted training is introduced to fully combine the advantages of anchor-based and anchor-free training modes.

7. A swimming pool drowning detection method based on RTMDet according to claim 6, characterized in that: In S4.1, the highest classification score and the most accurate bounding box are obtained by learning TAL through task alignment. The calculation method is as follows: S4.1.1: Use a high-order combination of classification scores and IoU to measure the degree of task alignment t, calculated as follows: t=s α ×u β Among them, s and u represent the classification score and IoU value respectively, and α and β are used to control the impact on the anchor alignment metric; S4.1.2: Use a normalized t to replace the binary label of the positive anchor point, calculated as follows: Among them, i represents the i-th anchor point among the positive anchor points corresponding to an object, represents the normalized t; S4.1.3: Focal loss is used for classification. The final loss function of the classification task is defined as follows: Where j represents the jth anchor in the negative anchor point, and γ is the focusing parameter; S4.1.4: The loss function for the regression task is defined as follows: Among them, b and Represents the predicted bounding box and the corresponding ground-truth box; S4.1.5: The final loss function is defined as follows: L loss =L cls +L reg .

8. A swimming pool drowning detection method based on RTMDet according to claim 6, characterized in that: The anchor frame assisted training is introduced in S4.2, which specifically includes: During the training phase, the anchor-assisted branch and the anchor-free branch perform loss calculations and gradient propagation respectively, and the auxiliary head is removed during inference.

Images