A method and system for intelligent identification of drowning persons based on deep learning

Abstract

The present invention relates to the field of image processing technology, and more specifically, to a method and system for intelligently identifying drowning persons based on deep learning, comprising: collecting swimming images of swimmers; inputting the swimming images into a trained classification model, and outputting the position status of the swimmers, wherein the position status of the swimmers includes being on the water surface or underwater; according to the number of key points of the swimmer's body exposed above the water surface, the time the swimmers are submerged underwater, and the movement speed of the swimmers. The present invention identifies swimmers suspected of drowning persons with multiple drowning features through a deep convolutional neural network. Furthermore, for swimmers suspected of drowning persons, a combination of multiple drowning feature recognition time superposition analysis is performed on them within a continuous observation time, and the drowning probability value of the swimmers is calculated. The swimmers are safely monitored according to the drowning probability value, thereby improving the accuracy of drowning identification of swimmers.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology. More specifically, the present invention relates to a method and system for intelligently identifying drowning persons based on deep learning. Background Art

[0002] As we all know, swimming pool safety involves many aspects and many safety technical directions, one of the most important of which is drowning prevention. For swimming pool operators, drowning is also the most important safety issue. Once a drowning accident occurs in a swimming pool, it will not only endanger the life safety of swimmers, but also affect the normal operation of the swimming pool, and even cause huge economic losses and reputation losses to the swimming pool. At present, there are many drowning prevention technologies, most of which use computer vision recognition technology, coupled with artificial intelligence systems, to determine whether a person is drowning.

[0003] In the related technology, for example, the Chinese patent application document with publication number CN115410273A discloses a drowning identification method, device and system. By performing a series of analyses on swimming pool pictures, it is possible to identify whether there are drowning people in the swimming pool, and effectively monitor the safety of people in the swimming pool. This solves the technical problem that the existing video surveillance system only collects swimming pool pictures but does not analyze the swimming pool pictures, and the analysis relies on manual work, which not only reduces the monitoring effect but also increases the workload of the monitoring personnel.

[0004] However, the current application of computer vision recognition technology in the field of drowning behavior recognition faces several limitations. First, some computer vision recognition methods rely on static image analysis, but such methods ignore the key time series information in continuous dynamic behavior. Secondly, although there are cameras with recognition algorithms to monitor people in real time, these recognition algorithms often use more traditional recognition algorithms, the accuracy of the recognition system is low, and the recognition results are not accurate enough. Summary of the invention

[0005] The present invention provides a method and system for intelligently identifying drowning persons based on deep learning, aiming to solve the problem that although related technologies use camera devices with recognition algorithms to monitor persons in real time, these recognition algorithms often apply more traditional recognition algorithms, the accuracy of the recognition system is low, and the recognition results are not precise enough.

[0006] In a first aspect, the present invention provides a method for intelligently identifying drowning persons based on deep learning, comprising: collecting swimming images of swimmers; inputting the swimming images into a trained classification model, and outputting the position status of the swimmer, wherein the position status of the swimmer includes being on the water surface or being underwater; calculating the drowning probability value of the target swimmer according to the number of key points of the swimmer's body exposed above the water surface, the submerged time of the swimmer and the moving speed of the swimmer, wherein the drowning probability value is negatively correlated with the number of key points of the body and the moving speed, and is positively correlated with the submerged time; and performing drowning detection on the swimmer according to the size of the drowning probability value.

[0007] Preferably, drowning detection is performed on the swimmer based on the size of the drowning probability value, including: in response to the drowning probability value being greater than a first threshold, the swimmer is at risk of drowning, and an alarm is issued; in response to the drowning probability value being less than the first threshold, the swimmer is not at risk of drowning, and no alarm is issued.

[0008] Preferably, according to the magnitude of the drowning probability value, drowning detection is performed on the swimmer, including: calculating the drowning probability value of the target swimmer , the calculation formula is: ; In the formula, is a constant, Indicates the time the swimmer is underwater. The number of key points above the swimmer's shoulders and shoulders that are above the water. is the moving speed of the swimmer in the current swimming image and the previous swimming image, represents the standard normalization function, is a negatively correlated exponential function.

[0009] Preferably, the drowning detection is performed on the swimmer according to the magnitude of the drowning probability value, including: calculating the movement speed of the swimmer, including: inputting the swimming image into a target detection model to obtain a bounding box of the swimmer; calculating the movement speed of the swimmer according to the coordinates of the upper left corner of the bounding box in the two adjacent frames of the image, the width and height of the bounding box , the calculation formula is: = ; In the formula, Represents the horizontal coordinate of the upper left corner of the bounding box in the previous swimming image, Represents the horizontal coordinate of the upper left corner of the bounding box in the current frame swimming image, Represents the ordinate of the upper left corner of the bounding box in the previous swimming image, Represents the ordinate of the upper left corner of the bounding box in the current frame swimming image, and Respectively represent the width and height of the bounding box in the previous swimming image, and Respectively represent the width and height of the bounding box in the swimming image of the current frame.

[0010] Preferably, drowning detection is performed on a swimmer according to the magnitude of the drowning probability value, including: calculating the movement speed of the swimmer, and also including: obtaining any human body key point of the swimmer in two adjacent frames of the swimming image; calculating the Euclidean distance between the human body key point in the current frame of the swimming image and the human body key point in the previous frame of the swimming image, and calculating the ratio of the Euclidean distance to the time interval between the two frames of swimming images before and after the acquisition, to obtain the movement speed of the swimmer.

[0011] Preferably, performing drowning detection on a swimmer according to the magnitude of the drowning probability value includes: further including: obtaining the body key points of the swimmer; inputting the swimming image into a body key point detection model, and outputting the body key points.

[0012] Preferably, drowning detection is performed on the swimmer based on the size of the drowning probability value, including: the submersion time of the swimmer underwater, including: in response to the time that the swimmer's mouth or nose is above the water surface being greater than a second threshold, recalculating the submersion time of the swimmer underwater.

[0013] Preferably, performing drowning detection on a swimmer according to the magnitude of the drowning probability value comprises: inputting the swimming image into a target detection model to obtain a bounding box of the swimmer, comprising: training the target detection model using a target detection algorithm.

[0014] In a second aspect of the present invention, a system for intelligently identifying drowning persons based on deep learning is also provided, comprising a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement any of the above-mentioned intelligent methods for intelligently identifying drowning persons based on deep learning.

[0015] Beneficial effects: Swimmers suspected of drowning with multiple drowning features are identified through deep convolutional neural networks. Furthermore, for swimmers suspected of drowning, multiple drowning feature recognition combined time superposition analysis is performed on them during continuous observation time, and the drowning probability value of the swimmer is calculated. The swimmer is safely monitored based on the drowning probability value, which improves the accuracy of swimmer drowning identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The following detailed description is read with reference to the accompanying drawings, which illustrate several embodiments of the present invention in an exemplary and non-limiting manner, and in which like or corresponding reference numerals represent like or corresponding parts, wherein:

[0017] Figure 1is a flow chart schematically illustrating calculation of a drowning probability value of a swimmer according to an embodiment of the present invention;

[0018] Figure 2 is a schematic diagram schematically showing a bounding box of a target detection model according to an embodiment of the present invention;

[0019] Figure 3 is a schematic diagram schematically illustrating target tracking according to an embodiment of the present invention;

[0020] Figure 4 FIG. 1 is a schematic diagram schematically showing the structure of a system according to an embodiment of the present invention.

[0021] Reference numerals

[0022] 1. Processor; 2. Communication bus; 3. Memory; 4. Communication interface. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0024] like Figure 1-Figure 3 As shown, the specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings.

[0025] S101: Collect swimming images of a swimmer.

[0026] Specifically, a camera can be used to take a photo of a swimmer swimming in a swimming pool, thereby obtaining an image of the swimmer. Since there are many detection targets in the image, it is necessary to find the detection target in the image. In this embodiment, the detection target is the swimmer. Therefore, it is necessary to process the image to detect the swimmers in the image, so that each swimmer has a corresponding bounding box, and the image containing the bounding box is used as the swimming image.

[0027] like Figure 2As shown, in one embodiment, the image can be processed by a trained target detection model to detect a specified type of target from the image, wherein the target refers to a swimmer. Specifically, the image is input into the target detection model, and a swimming image is output, wherein each swimmer in the swimming image corresponds to a bounding box. For acquiring the target detection model, it includes: training a target detection model using the YOLOv8 algorithm (target detection algorithm) and a public data set. Among them, YOLOv8 is a target detection algorithm based on deep learning, and its core idea is to extract features from the input image by training a deep convolutional neural network, and use the detection box and the position information on the feature map to detect the target.

[0028] In another embodiment, in order to improve the recall rate of detecting underwater swimmers, pictures of swimming pool scenes may be collected, and the positions of swimmers may be manually annotated for fine-tuning the YOLOv8 model trained based on a public dataset.

[0029] In one embodiment, since it is difficult to judge the state of a swimmer based on a single picture, an algorithm is required to perform a comprehensive analysis of continuous video frames. Therefore, the present embodiment involves target tracking technology, and target tracking here refers to: establishing associations between swimmers between continuous video frames, that is, determining which swimmers in the current frame are the same person as in the previous frame or frames. Among them, target tracking is performed using the ByteTrack algorithm (a tracking algorithm based on target detection). Specifically, the swimming image data of each swimmer from the target detection algorithm is received through a target tracking method, and the dynamic trajectory of each swimmer is tracked by integrating motion trajectory prediction and comparing and analyzing the swimmer's characteristic images, thereby obtaining the state of the swimmer on the timeline.

[0030] For example, Figure 3 As shown, two swimmers are detected in the swimming pool through the target detection algorithm, wherein one swimmer's ID is named 1 and the other swimmer's ID is named 2. Then, by using target tracking for the two swimmers, the positions of the two swimmers can be tracked in real time to obtain the movement trajectories of the swimmers.

[0031] In one embodiment, regarding the installation of cameras: at least one camera may be deployed around the swimming pool. In other embodiments, in order to make the detection area more comprehensive and avoid blind spots caused by one camera, multiple cameras may be deployed around the swimming pool to increase the detection area and improve the safety of subsequent detections.

[0032] S102: Detect the position status of the swimmer.

[0033] In one embodiment, the position state of the swimmer is divided into two types, one of which is that the position state of the swimmer is on the water surface, and the other is that the position state of the swimmer is underwater.

[0034] In one embodiment, the image classification algorithm SENet can be used to identify the position state of the swimmer and determine whether the swimmer is submerged in the water. Specifically, a large number of image samples of the swimmer on the water surface and underwater are collected, and each image is manually annotated to clearly mark whether the position state of the swimmer in the image is "water surface" or "underwater", and then the classification model is trained to obtain a trained classification model. Then the swimming image is input into the trained classification model, and the position state of the swimmer is output, wherein the position state of the swimmer includes being on the water surface or being underwater. So far, the submergence time of the swimmer can be counted by the above classification model. Specifically, when the underwater position state of the swimmer is identified, the duration of the swimmer's underwater position state is counted, and the longer the duration, the greater the risk of drowning of the swimmer. It should be noted that when the time when the swimmer's mouth or nose is on the water surface is greater than the second threshold, the submergence time of the swimmer under water is recalculated. Among them, the second threshold is 2S, and in other embodiments, the second threshold can be 1.3S, 1.5S, etc., and the second threshold can be adjusted manually.

[0035] S103: Calculate the drowning probability value of the swimmer.

[0036] Specifically, the drowning probability value of the target swimmer is calculated according to the number of key points of the swimmer's body exposed to the water surface, the submerged time of the swimmer and the moving speed of the swimmer, wherein the drowning probability value is negatively correlated with the number of key points of the body and the moving speed, and positively correlated with the submerged time;

[0037] In one embodiment, the acquisition of human key points includes: inputting the swimming image into a human key point detection model, and outputting human key points, wherein the human key points include left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle. The key points on the shoulder and above the shoulder are left shoulder, right shoulder, left ear, right ear, left eye, right eye and nose. Among them, the human key point detection model used to detect human key points belongs to the prior art. The human key point detection model is a model in the field of computer vision, which aims to identify and locate the key points of the human body in an image or video, usually including the key points of the head, limbs and other important parts. For example, the PoseNet model is used, which is a model for real-time posture estimation. It is a deep learning model based on convolutional neural network (CNN) for detecting and tracking the key points of the human body from the image of the camera.

[0038] In one embodiment, the drowning probability value of the target swimmer is calculated. , the calculation formula is: ; In the formula, is a constant, Indicates the time the swimmer is underwater. The number of key points above the swimmer's shoulders and shoulders that are above the water. is the moving speed of the swimmer in the current swimming image and the previous swimming image, represents the standard normalization function, is a negatively correlated exponential function, and the final calculated drowning probability value is A value between 0 and 1.

[0039] in, The larger the value of , the longer the swimmer stays underwater. The longer the swimmer stays underwater, the greater the risk of drowning, and the greater the drowning probability value. The larger the value of , the more shoulders and key points above the shoulders of the swimmer are exposed above the water surface, and the more parts of the swimmer's shoulders are exposed above the water surface. The mouth and nose are both parts above the shoulders, and the human body relies on the mouth and nose to maintain oxygen absorption. Therefore, when the swimmer's shoulders are exposed above the water surface, the risk of drowning is smaller, and the drowning probability value is smaller. For the movement speed V of the swimmer in the current frame swimming image and the previous frame swimming image, in the case of normal swimming, the swimmer will move at a certain speed and will not be stationary. Therefore, the smaller the value of the movement speed V, the greater the risk of drowning of the swimmer, and the greater the drowning probability value.

[0040] In one embodiment, calculating the movement speed of a swimmer includes: inputting the swimming image into a target detection model to obtain a bounding box of the swimmer; calculating the movement speed of the swimmer according to the coordinates of the upper left corner of the bounding box in the two adjacent frames of the image, the width and height of the bounding box; , the calculation formula is: = ; In the formula, Represents the horizontal coordinate of the upper left corner of the bounding box in the previous swimming image, Represents the horizontal coordinate of the upper left corner of the bounding box in the current frame swimming image, Represents the ordinate of the upper left corner of the bounding box in the previous swimming image, Represents the ordinate of the upper left corner of the bounding box in the current frame swimming image, and Respectively represent the width and height of the bounding box in the previous swimming image, and Represent the width and height of the bounding box in the swimming image of the current frame, respectively. It should be noted that the purpose of using this method to calculate the formula is to take into account the average scale of the swimmer's moving speed in the image, which may be affected by its size in the image.

[0041] In another embodiment, calculating the movement speed of the swimmer also includes: obtaining any human body key point of the swimmer in two adjacent frames of the swimming image; calculating the Euclidean distance between the human body key point in the current frame of the swimming image and the human body key point in the previous frame of the swimming image, and calculating the ratio of the Euclidean distance to the time interval between the two frames of swimming images before and after the acquisition, to obtain the movement speed of the swimmer.

[0042] S104: Perform drowning detection on the swimmer according to the magnitude of the drowning probability value.

[0043] In one embodiment, if the drowning probability value is greater than a first threshold, the swimmer is at risk of drowning, and an alarm is triggered; if the drowning probability value is less than the first threshold, the swimmer is not at risk of drowning, and no alarm is triggered. In this embodiment, the first threshold is 0.6, and in other embodiments, the first threshold may be 0.5 or 0.55, etc., and may be adjusted according to specific implementation scenarios.

[0044] In one embodiment, if the overall monitoring safety of swimmers is desired to be higher, the first threshold is set to a smaller value, for example, the first threshold is set to 0.2 or 0.25, which can further improve the safety of drowning identification.

[0045] According to the above steps, the method for identifying multiple drowning features proposed in this paper identifies swimmers suspected of drowning with multiple drowning features through a deep convolutional neural network. Furthermore, for swimmers suspected of drowning, a combined time superposition analysis of multiple drowning feature recognition is performed within a continuous observation time, and the drowning probability value of the swimmer is calculated. The swimmer is safely monitored according to the drowning probability value, which improves the accuracy of swimmer drowning identification.

[0046] In another embodiment, whether the swimmer is at risk of drowning can also be determined based on any one or any combination of two of the swimmer's submerged time, the swimmer's shoulders exposed above the water, the number of key points above the shoulders, and the swimmer's moving speed.

[0047] Exemplarily, when the swimmer's submerged time under water is greater than the third threshold, it indicates that the swimmer is at risk of drowning, and an alarm is issued, wherein the third threshold may be 4S or 5S, and the size of the third threshold may be adjusted manually. When the swimmer's shoulders and key points above the shoulders are not exposed above the water, it is marked as the swimmer is in a drowning state, and an alarm is issued. When the swimmer's moving speed is less than the fourth threshold, it indicates that the swimmer is at risk of drowning, and an alarm is issued. Among them, the size of the third threshold may be adjusted manually.

[0048] The present invention also provides a drowning person intelligent identification system based on deep learning. Figure 4 As shown, the system includes a processor 1 and a memory 3, wherein the memory 3 stores computer program instructions. When the computer program instructions are executed by the processor 1, a method for intelligently identifying drowning persons based on deep learning according to the first aspect of the present invention is implemented.

[0049] The system also includes other components well known to those skilled in the art, such as a communication bus 2 and a communication interface 4. The configuration and functions of these components are known in the art and will not be described in detail herein.

[0050] In the present invention, the aforementioned memory 3 may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus or device. For example, a computer-readable storage medium may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc., or any other medium that may be used to store the required information and may be accessed by an application, a module, or both. Any such computer storage medium may be part of a device or accessible or connectable to a device. Any application or module described in the present invention may be implemented using computer-readable / executable instructions that may be stored or otherwise maintained by such a computer-readable medium.

[0051] In the description of this specification, "plurality" or "several" means at least two, such as two, three or more, etc., unless otherwise clearly and specifically defined.

[0052] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0053] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent application. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for intelligently identifying drowning persons based on deep learning, characterized in that: include: collecting swimming images of swimmers; Inputting the swimming image into a trained classification model to output a position state of the swimmer, wherein the position state of the swimmer includes being on the water surface or being under the water; Calculate the drowning probability of the target swimmer according to the number of key points of the swimmer's body exposed above the water surface, the submerged time of the swimmer, and the swimmer's moving speed, wherein the drowning probability is negatively correlated with the number of key points of the body and the moving speed, and positively correlated with the submerged time; According to the magnitude of the drowning probability value, drowning detection is performed on the swimmer.

2. The method for intelligently identifying drowning persons based on deep learning according to claim 1 is characterized in that: According to the magnitude of the drowning probability value, a drowning detection is performed on the swimmer, including: In response to the drowning probability value being greater than a first threshold value, the swimmer is at risk of drowning, and an alarm is issued; In response to the drowning probability value being less than the first threshold, the swimmer is not at risk of drowning and no alarm is issued.

3. The method for intelligently identifying drowning persons based on deep learning according to claim 1 is characterized in that: Calculate the drowning probability value P of the target swimmer, the calculation formula is: ； Where a is a constant, T represents the submerged time of the swimmer, G is the swimmer's shoulder exposed above the water surface and the number of key points above the shoulder, V is the movement speed of the swimmer in the current swimming image and the previous swimming image, norm[] represents the standard normalization function, and exp() is the negative correlation exponential function.

4. The method for intelligently identifying drowning persons based on deep learning according to claim 1 is characterized in that: Calculate the swimmer's movement speed, including: Inputting the swimming image into an object detection model to obtain a bounding box of the swimmer; According to the coordinates of the upper left corner of the bounding box in two adjacent frames, the width and height of the bounding box, the moving speed V of the swimmer is calculated. The calculation formula is: = ； Wherein, X1 represents the abscissa of the upper left corner of the bounding box in the previous swimming image, X2 represents the abscissa of the upper left corner of the bounding box in the current swimming image, Y1 represents the ordinate of the upper left corner of the bounding box in the previous swimming image, Y2 represents the ordinate of the upper left corner of the bounding box in the current swimming image, H1 and W1 represent the width and height of the bounding box in the previous swimming image, respectively, and H2 and W2 represent the width and height of the bounding box in the current swimming image, respectively.

5. The method for intelligently identifying drowning persons based on deep learning according to claim 1 is characterized in that: Calculate the swimmer's movement speed, also including: Acquire any key point of the swimmer in two adjacent frames of the swimming image; The Euclidean distance between the key points of the human body in the current swimming image and the key points of the human body in the previous swimming image is calculated, and the ratio of the Euclidean distance to the time interval between the two swimming image frames before and after the acquisition is calculated to obtain the movement speed of the swimmer.

6. The method for intelligently identifying drowning persons based on deep learning according to claim 3 is characterized in that: It also includes obtaining key points of the swimmer's body; The swimming image is input into a human body key point detection model to output the human body key points.

7. The method for intelligently identifying drowning persons based on deep learning according to claim 1 is characterized in that: The time a swimmer is underwater, including: In response to the time that the swimmer's mouth or nose is above the water surface being greater than a second threshold, the submerged time that the swimmer is underwater is recalculated.

8. The method for intelligently identifying drowning persons based on deep learning according to claim 4 is characterized in that: The swimming image is input into the object detection model to obtain the bounding box of the swimmer, including: Use the object detection algorithm to train the object detection model.

9. A drowning person intelligent identification system based on deep learning, characterized in that: It includes a processor and a memory, the memory stores a computer program, and the processor executes the computer program to implement the deep learning-based intelligent identification method for drowning persons as described in any one of claims 1 to 8.

Images