Baby dangerous action detection and recognition method based on pose estimation and improved YOLOX_tiny

Abstract

The application discloses a baby dangerous action detection and recognition method based on pose estimation and improved YOLOX_tiny, first, video frames are acquired from a camera, and the input video frames are preprocessed; second, the preprocessed video frames are input into a pose estimation algorithm and an improved YOLOX_tiny target detection algorithm based on model pruning and fusion of a spatial attention mechanism to obtain the number of skeleton points, the target region of interest and the category detection result data in each frame of picture, wherein the target detection algorithm is trained by using a self-built data set and a transfer learning mode; then, the obtained data is processed and fused to improve the accurate recognition ability of different dangerous actions; finally, the processed and fused data is used to design rules to judge the normal and dangerous action states of babies and issue alarms. The application has important significance for protecting the safe and healthy growth of babies, reducing the care pressure of newborn parents, and has high recognition accuracy and low false detection rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pose estimation and target detection based on computer vision, in particular to a baby dangerous action detection and recognition method based on pose estimation and improved YOLOX_tiny. BACKGROUND

[0002] In the field of action recognition algorithms, it can be generally divided into action recognition algorithms based on video sequences and action recognition algorithms based on skeleton point sequences. However, the research object of the action classification and recognition algorithm at home and abroad at present is mainly adult, and there is a lack of image or video dataset for babies. Due to the difference in body size and limb movement between babies and adults, the existing model does not have robustness in the baby action recognition task. Since babies belong to a special group, the action recognition algorithm based on video sequences and the action recognition algorithm based on skeleton point sequences require high data acquisition and labeling cost. Moreover, in the face of complex action recognition of multiple categories, most algorithms have poor real-time performance due to the complexity of their own structure and the limitation of hardware configuration. SUMMARY

[0003] The purpose of the present application is to provide a baby dangerous action detection and recognition method based on pose estimation and improved YOLOX_tiny to solve the problems raised in the background.

[0004] To achieve the above purpose, the present application provides the following technical scheme:

[0005] The baby dangerous action detection and recognition method based on pose estimation and improved YOLOX_tiny comprises the following steps:

[0006] Obtain video frames from a camera, and pre-process the input video frames;

[0007] Input the pre-processed video frames into a pose estimation algorithm to obtain the number of skeleton points in each frame of picture, and input them into an improved YOLOX_tiny target detection algorithm based on model pruning and fusion spatial attention mechanism to obtain the target region and category detection result data of each frame of picture of interest, wherein the target detection algorithm is trained using a self-built dataset and using a transfer learning method;

[0008] The pose estimation algorithm uses a lightweight openpose pose estimation algorithm to obtain the number of key skeleton points of the baby in each frame.

[0009] The model pruning method is obtained by removing the bottom-up structure in the path aggregation feature pyramid network in the original YOLOX_tiny and only retaining a top-down structure. The model pruning reduces the overall parameter quantity of the model, thereby improving the real-time performance of the algorithm model in practical application.

[0010] For the spatial attention mechanism, firstly, it includes two pooling layers: average pooling and max pooling. The pooling structure aggregates the input feature map along the channel direction, resulting in two channels, each with a channel count of 1, and whose height and width are the same as the input feature map. Figure One First, the two feature maps are concatenated along the channel direction. Then, a convolution with a kernel of 1 is applied to change the number of channels. Next, a sigmoid activation function is used to increase non-linearity and prevent overfitting. Finally, the feature map containing spatial feature information obtained by the sigmoid activation function is multiplied with the input feature map to obtain the final output feature.

[0011] The spatial attention mechanism described is positioned in YOLOX_tiny by adding a spatial attention module after each CSPlayer layer in the YOLOX_tiny backbone network CSPDarknet in order to improve the algorithm's attention to the key features of the target of interest and thus improve the overall detection accuracy.

[0012] The obtained data is processed and fused to improve the ability to accurately identify different dangerous actions;

[0013] Data processing and fusion methods include:

[0014] The algorithm for recognizing dangerous actions of infants and young children is initialized using the first N frames of video data to determine whether the infant or young child is currently sleeping with a blanket or not.

[0015] The pose estimation algorithm is used to detect and save the number K of bone points in each of the first N frames. i Go to list A;

[0016] The number of skeleton points K in each frame is detected using a pose estimation algorithm. i Go to list B;

[0017] Using an improved target detection algorithm, the target of interest is detected in each frame, and the detection results of each frame are saved to list D in sequence. When the frame detection result contains climbing, add 1 to list C, otherwise add 0. Similarly, when the frame detection result contains the lower body in a lying position, add 1 to list E, otherwise add 0.

[0018] For list A, when the length reaches N, stop updating and save it for later use;

[0019] For lists B, C, D, and E, when the length is greater than N, delete the first element in lists B, C, D, and E, update the list elements, and iterate until the list length is always N.

[0020] By utilizing processed and fused data, rules are designed to determine the normal and dangerous behavioral states of infants and toddlers and to issue alarms. The process includes:

[0021] When the total of the first N frames of skeleton points stored in list A is greater than the set threshold K, climbing and loss monitoring is performed, otherwise kicking and loss monitoring is performed.

[0022] For climbing and loss monitoring:

[0023] Judgment 1: When the number of 0 in list B and the number of empty lists in list D are both N, it is judged that the baby is lost.

[0024] Judgment 2: When the number of 1 in list C is greater than N / 2, it is judged that the baby is climbing.

[0025] Judgment 3: Except for judgments 1 and 2, it is considered as a normal state.

[0026] For kicking and loss monitoring:

[0027] Judgment 4: When the number of 0 in list B and the number of empty lists in list D are both N, it is judged that the baby is lost.

[0028] Judgment 5: When the first element in list E is 0 and the last element is 1, it is judged that the baby is kicking the quilt.

[0029] Judgment 6: Except for judgments 4 and 5, it is considered as a normal state.

[0030] Among them, the threshold N and the pose estimation are used to strengthen the data joint judgment, and the false detection and false report probability is reduced.

[0031] As a preferred scheme of the present application: the target detection algorithm related data set construction mainly includes data collection, labeling and division, and the detailed steps are:

[0032] a1, collect state pictures of infants in indoor environment, including standing, lying, climbing, sitting and crawling five state pictures;

[0033] a2, label the collected data, and the labeling content includes upper body, lower body and whole body in standing state, upper body, lower body and whole body in lying state, whole body in climbing, whole body in sitting, whole body in crawling, head, 10 kinds of labeling content;

[0034] a3, divide the data set into training set, verification set and test set, the proportion is 8:1:1.

[0035] Compared with the prior art, the present application has the advantages that: the present application has high recognition accuracy, low false detection rate and high robustness; it has important significance in reducing the stress of newborn parents; and the popularization and application of the present application can effectively protect the safe and healthy growth of infants. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a flowchart of the method of the present application;

[0037] Figure 2 is a detailed flowchart of the present application;

[0038] Figure 3 is the original structure diagram of YOLOX_tiny algorithm;

[0039] Figure 4 is the structure diagram of the improved YOLOX_tiny algorithm;

[0040] Figure 5 is a structure diagram of a spatial attention mechanism. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0042] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0043] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0044] Please refer to Figure 1 In the embodiments of the present application, the infant dangerous action detection and recognition method based on pose estimation and improved YOLOX_tiny includes the following steps:

[0045] A1, obtaining video frames from a camera, and pre-processing the input video frames;

[0046] A2, input the pre-processed video frame into the pose estimation algorithm to obtain the number of skeletal points in each frame of picture, and input into the YOLOX_tiny target detection algorithm improved based on model pruning and fusion spatial attention mechanism to obtain the target region of interest and class detection result data of each frame of picture, wherein the target detection algorithm is trained using self-built dataset and using transfer learning method;

[0047] A3, processing and fusing the obtained data to improve the accurate recognition ability of different dangerous actions;

[0048] A4, using the processed and fused data, designing rules to judge the normal and dangerous action state of infants and children and issuing alarms.

[0049] Among them, the pose estimation algorithm in step A2 uses a lightweight openpose pose estimation algorithm to obtain the number of key skeletal points of infants and children in each frame.

[0050] The model pruning method is obtained by removing the bottom-up structure in the PathAggregation and Feature Pyramid Network (PAFPN) in the original YOLOX_tiny and only retaining a top-down structure. The model pruning reduces the overall parameter quantity of the model, thereby improving the real-time performance of the algorithm model in practical application.

[0051] For the spatial attention mechanism, first, the attention mechanism includes two pooling layers, average pooling and maximum pooling, and the pooling structure aggregates the input feature map along the channel direction to obtain two feature maps with channel number 1 and height, width and input feature Figure One The feature map is the same; second, the two feature maps are concatenated in the channel direction; third, a convolution kernel of 1 is used to change the channel number; then, a sigmoid activation function is used to increase nonlinearity and prevent overfitting; finally, the feature map containing spatial feature information obtained by the sigmoid activation function is multiplied by the input feature map to obtain the final output feature.

[0052] The position of the spatial attention mechanism in YOLOX_tiny is to improve the attention of the key features of the target of interest to improve the overall detection accuracy. The spatial attention (SA) module is added after each layer of CSPlayer in the CSPDarknet of the YOLOX_tiny backbone network.

[0053] The construction of the related dataset of the target detection algorithm mainly includes data collection, labeling and division, and the detailed steps are as follows:

[0054] a1, collect the state pictures of infants in indoor environment, including standing, lying, climbing, sitting and crawling;

[0055] a2, label the collected data, the labeling content including upper body, lower body and whole body in standing state, upper body, lower body and whole body in lying state, whole body in climbing, whole body in sitting, whole body in crawling, head, 10 kinds of labeling content;

[0056] a3, divide the data set into training set, validation set and test set, the proportion is 8:1:1.

[0057] In step A3, the data processing and fusion method includes:

[0058] The infant dangerous action recognition algorithm is initialized by using the first N frames of video data to determine whether the infant is in a covered sleeping state or a non-sleeping state;

[0059] The pose estimation algorithm is used to detect and save the number K of bone points in each frame of the first N frames i to list A;

[0060] The pose estimation algorithm is used to detect the number K of bone points in each frame i to list B;

[0061] An improved target detection algorithm is used to detect each frame of interest target, and the detection results of each frame are saved to list D in turn, when the frame detection result contains climbing, add 1 to list C, otherwise add 0, similarly, when the frame detection result contains lying state lower body, add 1 to list E, otherwise add 0;

[0062] For list A, when the length reaches N, stop updating and save as standby;

[0063] For lists B, C, D and E, when the length is greater than N, delete the first element in lists B, C, D and E, and update the list elements, iterate and run, always keep the list length as N.

[0064] The main process of step A4 includes:

[0065] When the total number of bone points stored in list A for the first N frames is greater than the set threshold K, climbing and loss monitoring is performed, otherwise kicking the quilt and loss monitoring is performed.

[0066] For climbing and loss monitoring:

[0067] Judgment 1, when the number of 0 in list B and the number of empty list in list D are both N, it is judged that the baby is lost;

[0068] Judgment 2, when the number of 1 in list C is greater than N / 2, it is judged that the baby is climbing;

[0069] Judgment 3, in addition to judgments 1 and 2, it is considered as a normal state.

[0070] For kicking the quilt and loss monitoring:

[0071] Judgment 4, when the number of 0 in list B and the number of empty list in list D are both N, it is judged that the baby is lost;

[0072] Judgment 5, when the first element in list E is 0 and the last element is 1, it is judged that the baby is kicking the quilt;

[0073] Judgment 6, in addition to judgments 4 and 5, it is considered as a normal state;

[0074] Among them, the threshold N and the pose estimation enhancement are used to jointly judge the data, and the false detection and false report probability is reduced.

[0075] It is apparent to those skilled in the art that the present application is not limited to the details of the foregoing exemplary embodiments, and that the present application can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and not restrictive, the scope of the present application being defined by the appended claims rather than the foregoing description, and it is intended that all changes falling within the meaning and range of equivalency of the elements of the claims are embraced therein. Any reference signs in the claims should not be construed as limiting the claims to the figures in which the reference signs are used.

[0076] In addition, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be appropriately combined to form other embodiments that those skilled in the art can understand.

Claims

1. A method for detecting and recognizing dangerous movements in infants and toddlers based on pose estimation and improved YOLOX_tiny, characterized in that, Includes the following steps: A1. Acquire video frames from the camera and preprocess the input video frames; A2. Input the preprocessed video frames into the pose estimation algorithm to obtain the number of bone points in each frame, and input them into the YOLOX_tiny object detection algorithm improved based on model pruning and fusion spatial attention mechanism to obtain the target region of interest and category detection results data for each frame. The object detection algorithm is trained using a self-built dataset and transfer learning. A3. Process and fuse the obtained data to improve the ability to accurately identify different dangerous actions; A4. Utilize the processed and fused data to design rules to determine the normal and dangerous movement states of infants and young children and issue alarms accordingly; The data processing and fusion method in step A3 includes: The algorithm for recognizing dangerous actions of infants and young children is initialized using the first N frames of video data to determine whether the infant or young child is currently sleeping with a blanket or not. The pose estimation algorithm is used to detect and save the number K of bone points in each of the first N frames. i Go to list A; The number of skeleton points K in each frame is detected using a pose estimation algorithm. i Go to list B; Using an improved target detection algorithm, the target of interest is detected in each frame, and the detection results of each frame are saved to list D in sequence. When the frame detection result contains climbing, add 1 to list C, otherwise add 0. Similarly, when the frame detection result contains the lower body in a lying position, add 1 to list E, otherwise add 0. For list A, when the length reaches N, stop updating and save it for later use; For lists B, C, D, and E, when the length is greater than N, delete the first element in lists B, C, D, and E, update the list elements, and iterate until the list length is always N. The process of step A4 includes: When the sum of the first N frame skeleton points stored in list A is greater than the set threshold K, climbing and loss detection are performed; otherwise, kicking the blanket and loss detection are performed. For climbing and loss monitoring: Judgment 1: When the number of occurrences of 0 in list B and empty lists in list D are both N, it is determined that the baby is lost. Judgment 2: When the number of occurrences of 1 in list C is greater than N / 2, it is determined that the baby is climbing; Judgment 3: Except for judgments 1 and 2, it is considered a normal state; Regarding the monitoring of kicking off blankets and loss: Judgment 4: When the number of occurrences of 0 in list B and empty lists in list D are both N, it is determined that the baby is lost. Judgment 5: When the first element of list E is 0 and the last element is 1, it is determined that the baby is kicking off the blanket. Judgment 6: Except for judgments 4 and 5, it is considered a normal state; The use of threshold N and pose estimation enhances the joint judgment of data, reducing the probability of false detection and false alarm.

2. The method for detecting and recognizing dangerous movements of infants and young children based on pose estimation and improved YOLOX_tiny according to claim 1, characterized in that, The pose estimation algorithm in step A2 uses the lightweight OpenPose pose estimation algorithm to obtain the number of key skeletal points of the infant's body in each frame.

3. The method for detecting and recognizing dangerous movements of infants and young children based on pose estimation and improved YOLOX_tiny according to claim 2, characterized in that, The model pruning method in step A2 is obtained by removing the bottom-up structure from the path aggregation feature pyramid network in the original YOLOX_tiny and keeping only one top-down structure. By pruning the model, the overall number of model parameters is reduced, thereby improving the real-time performance of the algorithm model in practical applications.

4. The method for detecting and recognizing dangerous movements of infants and young children based on pose estimation and improved YOLOX_tiny according to claim 3, characterized in that, For the spatial attention mechanism in step A2: First, the attention mechanism includes two pooling layers: average pooling and max pooling. The pooling structure aggregates the input feature map along the channel direction to obtain two feature maps with 1 channel and the same height and width as the input feature map. Secondly, the two feature maps are concatenated in the channel direction; Again, the number of channels is changed by performing a convolution with a kernel of 1; Then, the sigmoid activation function is used to add non-linearity and prevent overfitting; Finally, the feature map containing spatial feature information obtained by the sigmoid activation function is multiplied with the input feature map to obtain the final output feature.

5. The method for detecting and recognizing dangerous movements of infants and young children based on pose estimation and improved YOLOX_tiny according to claim 4, characterized in that, The spatial attention mechanism in YOLOX_tiny is positioned such that, in order to improve the algorithm's attention to the key features of the target of interest and thus improve the overall detection accuracy, a spatial attention module is added after each CSPlayer in the YOLOX_tiny backbone network CSPDarknet.

6. The method for detecting and recognizing dangerous movements of infants and young children based on pose estimation and improved YOLOX_tiny according to claim 5, characterized in that, The construction of the dataset related to the target detection algorithm includes data collection, annotation, and partitioning. The detailed steps are as follows: a1. Collect pictures of infants and toddlers in indoor environments, including pictures of five states: standing, lying down, climbing, sitting, and crawling. a2. Label the collected data. The labeling content includes the upper body, lower body and whole body in an upright position, the upper body, lower body and whole body in a lying position, the whole body in a climbing position, the whole body in a sitting position, the whole body in a crawling position, and the head. There are 10 types of labeling content. a3. Divide the dataset into a training set, a validation set and a test set in a ratio of 8:1:1.

Images