A sleeping post identification method, device, equipment and medium

Abstract

The present application discloses a method, device, equipment and medium for identifying a human body in a to-be-identified image, and relates to the field of image recognition technology. The method comprises: obtaining a head-shoulder area map corresponding to a human body target in a to-be-identified image, and determining a target Gaussian heat map corresponding to the head-shoulder area map; extracting semantic feature information of the head-shoulder area map and spatial feature information of the target Gaussian heat map; performing feature fusion of the semantic feature information and the spatial feature information, and judging whether the human body target in the to-be-identified image is in a sleeping state based on the fused feature information. Through the above scheme, when performing sleeping state identification, it is only necessary to judge whether the human body is sleeping by the head-shoulder area features, and it is possible to fully utilize the information in the image by extracting the semantic features of the head-shoulder area map and the spatial features of the target Gaussian heat map and by feature fusion. This can solve the problem of target occlusion during sleeping state identification and improve the accuracy of sleeping state identification.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a sleeping post recognition method, device, equipment and medium. Background Art

[0002] Sleeping on the job can easily lead to serious catastrophic accidents. In order to reduce the probability of safety accidents, users need to correct the sleeping behavior of on-the-job personnel in a timely manner. At present, most companies have achieved full coverage of surveillance videos, but factories, workshops, construction sites and other scenes have a wide field of view and a large flow of people. If you rely solely on the naked eye to monitor, it will not only consume manpower but also be difficult to stay alert for a long time. With the development of machine vision, deep learning methods can be used to detect sleeping behavior under video surveillance, which can greatly improve efficiency while reducing the burden of manpower.

[0003] At present, the sleeping guard detection methods based on deep learning are mainly divided into two categories: target detection and human posture estimation. However, the human posture estimation method will fail to detect some key points when the target is sideways or severely blocked, resulting in poor sleeping guard recognition based on human key points. On the other hand, if the detection results are directly input into the convolutional neural network built based on ShuffleNet v2 to determine whether the person is sleeping, due to the diversity of sleeping postures, camera angles, scenes, etc., it is difficult to classify directly based on human images, and the classification accuracy will be low. Therefore, there are problems of poor recognition effect and low classification accuracy in the sleeping guard recognition process that need to be further solved. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment and medium for identifying sleeping guards, which can solve the problems of poor recognition effect and low classification accuracy in the process of sleeping guard identification. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a sleeping post identification method, comprising:

[0006] Obtain a head-shoulder region map corresponding to a human target in a to-be-recognized image, and determine a target Gaussian heat map corresponding to the head-shoulder region map;

[0007] Extracting semantic feature information of the head and shoulder area map and spatial feature information of the target Gaussian heat map;

[0008] The semantic feature information is fused with the spatial feature information, and based on the fused feature information, it is determined whether the human target in the to-be-identified picture is in a sleeping state.

[0009] Optionally, the step of obtaining a head and shoulder area map corresponding to a human target in a to-be-recognized image includes:

[0010] The image to be identified is input into the trained human head and shoulder detection model to obtain a head and shoulder area map corresponding to the human target in the image to be identified; the human head and shoulder detection model is a model built based on the YOLO v5 network, and the Neck in the YOLO v5 network adopts a BiFPN structure.

[0011] Optionally, determining a target Gaussian heat map corresponding to the head and shoulders area map includes:

[0012] Input the image to be identified into the trained key point detection model to obtain the human body key points of the image to be identified, and construct an original Gaussian heat map corresponding to the image to be identified based on the human body key points of the image to be identified;

[0013] A Gaussian hot spot corresponding to the head-shoulder area map is determined from the original Gaussian hot spot map to obtain a target Gaussian hot spot map corresponding to the head-shoulder area map.

[0014] Optionally, also include:

[0015] Acquire a target picture group including the picture to be identified and several pictures adjacent to the picture to be identified, and determine whether the picture to be identified includes a static human target by detecting changes in key point positions between different adjacent pictures in the target picture group;

[0016] Accordingly, the step of obtaining a head and shoulder area map corresponding to a human target in a to-be-recognized image includes:

[0017] When the image to be identified contains a static human target, a head and shoulder area map corresponding to the static human target in the image to be identified is obtained.

[0018] Optionally, also include:

[0019] Construct a human posture estimation model based on the Shuffle network and the HRNet network;

[0020] Accordingly, the step of inputting the image to be identified into the trained key point detection model to obtain the human key points of the image to be identified includes:

[0021] The image to be identified is input into the trained human posture estimation model, so as to perform human posture estimation on the image to be identified through the human posture estimation model, and obtain the key points of the human body detected in the human posture estimation process.

[0022] Optionally, extracting the semantic feature information of the head-shoulder area map and the spatial feature information of the target Gaussian heat map, fusing the semantic feature information with the spatial feature information, and judging whether the human target in the to-be-identified picture is in a sleeping state based on the fused feature information includes:

[0023] Through the trained state classification model, the semantic feature information of the head and shoulder area map and the spatial position relationship between different human key points corresponding to the target Gaussian heat map are extracted, and the semantic feature information and the spatial position information are feature fused. Then, the fused feature information is used to classify the state of the human target in the image to be identified to obtain the corresponding human state category; the human state category includes a category corresponding to the sleeping state and a category corresponding to the non-sleeping state.

[0024] Optionally, also include:

[0025] Using the GhostNet network as the basic network and the ArcFace function as the loss function, constructing the state classification model;

[0026] Among them, the GhostNet network in the state classification model is used to extract the semantic feature information of the head and shoulder area map and the spatial position relationship between different human body key points corresponding to the target Gaussian heat map.

[0027] In a second aspect, the present application discloses a sleeping post identification device, comprising:

[0028] A head-shoulder region determination module is used to obtain a head-shoulder region map corresponding to a human target in a to-be-identified image, and determine a target Gaussian heat map corresponding to the head-shoulder region map;

[0029] A feature information extraction module, used to extract the semantic feature information of the head and shoulder area map and the spatial feature information of the target Gaussian heat map;

[0030] The state judgment module is used to perform feature fusion on the semantic feature information and the spatial feature information, and judge whether the human target in the to-be-identified picture is in a sleeping state based on the fused feature information.

[0031] In a third aspect, the present application discloses an electronic device, comprising:

[0032] Memory, used to store computer programs;

[0033] A processor is used to execute the computer program to implement the steps of the sleeping post identification method disclosed above.

[0034] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the sleeping post identification method as disclosed above are implemented.

[0035] When performing sleeping post recognition, the present application first obtains the head and shoulder area map corresponding to the human target in the image to be recognized, and determines the target Gaussian heat map corresponding to the head and shoulder area map, then extracts the semantic feature information of the head and shoulder area map and the spatial feature information of the target Gaussian heat map, and finally fuses the semantic feature information with the spatial feature information, and judges whether the human target in the image to be recognized is in a sleeping post state based on the fused feature information. It can be seen that when performing sleeping post recognition, the present application obtains the head and shoulder area map corresponding to the human target and the corresponding target Gaussian heat map, and extracts features from the head and shoulder area map and the target Gaussian heat map respectively and then fuses features, and judges whether the corresponding human target is sleeping on the post based on the result after feature fusion. Therefore, in the present application, when performing sleeping post recognition, it is only necessary to judge through the head and shoulder area features, thereby avoiding the problem of poor sleeping post recognition due to the failure to detect some key points when the target is sideways or severely obscured; on the other hand, by extracting the semantic features of the head and shoulder area map and the spatial features of the target Gaussian heat map and fusion of the features, various information in the image is fully utilized, thereby improving the accuracy of sleeping post recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0037] Figure 1 A flow chart of a sleeping post identification method provided for this application;

[0038] Figure 2 A schematic diagram of the occlusion of some body structures in the scene of a person sleeping on the post provided for this application;

[0039] Figure 3 A specific sleep post identification method flow chart provided for this application;

[0040] Figure 4 A specific sleep post identification method flow chart provided for this application;

[0041] Figure 5 A schematic diagram of the dynamic target filtering process provided for this application;

[0042] Figure 6 Schematic diagram of head and shoulders area detection provided for this application;

[0043] Figure 7 Schematic diagram of the regional relationship between the head and shoulders area map and the target Gaussian heat map provided for this application;

[0044] Figure 8 A schematic diagram of the sleep post process judgment provided for this application;

[0045] Fig. 9 A schematic diagram of a specific sleeping post identification method provided for this application;

[0046] Fig.10 A schematic diagram of the structure of a sleeping post identification device provided in this application;

[0047] Fig.11 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] At present, the sleeping post detection methods based on deep learning are mainly divided into two types of methods: target detection and human posture estimation. However, the human posture estimation method, on the one hand, will fail to detect some key points when the target is sideways or severely blocked, resulting in poor sleeping post recognition based on human key points. On the other hand, if the detection results are directly input into the convolutional neural network built based on ShuffleNet v2 to determine whether the person is sleeping, due to the diversity of sleeping postures, camera angles, scenes, etc., it is difficult to classify directly based on human images, and the classification accuracy will be low. To this end, the present application provides a sleeping post recognition method that can effectively solve the problems of poor recognition of sleeping post behavior and low classification accuracy during the sleeping post recognition process.

[0050] The embodiment of the present invention discloses a method for identifying a sleeping post, see Figure 1 As shown, the method includes:

[0051] Step S11: obtaining a head-shoulder region map corresponding to a human target in a to-be-recognized image, and determining a target Gaussian heat map corresponding to the head-shoulder region map.

[0052] In this embodiment, a head-shoulder region map of a human target in a to-be-identified image is obtained, wherein the head-shoulder region map refers to a small image of the human head-shoulder region, including but not limited to an RGB image of the head-shoulder region. After obtaining the head-shoulder region map corresponding to the human target in the to-be-identified image, a target Gaussian heat map corresponding to the head-shoulder region map is determined. It is understandable that, in the actual monitoring process of sleeping guard detection, there will be problems of human body occlusion, such as Figure 2 As shown, the sleeping guard recognition effect based on the overall key point detection of the human body is not good. In this embodiment, the head and shoulder area map is used to avoid the problem of poor sleeping guard recognition effect caused by easy occlusion of other limb key points.

[0053] Step S12: extracting semantic feature information of the head and shoulder area map and spatial feature information of the target Gaussian heat map.

[0054] In this embodiment, the semantic feature information of the head and shoulder area map and the spatial position relationship between different human key points corresponding to the target Gaussian heat map are extracted. If the head and shoulder posture image is directly used for classification, the network model will indiscriminately extract the features and background information of various parts of the human body, ignoring the spatial information contained in the human body structure. In order to allow the classification model to extract more posture information, spatial features are extracted from the target Gaussian heat map. First, the gradient value of each Gaussian hotspot in the above target Gaussian heat map is calculated by back propagation, and the average value is taken. The average value is used as the weight value of the spatial information feature map obtained after spatial extraction and multiplied by the feature value of each point in the above feature map. The feature maps of the same layer are added and the average value is taken to obtain a single feature map for maximum and minimum normalization to obtain the spatial feature information of the target Gaussian heat map.

[0055] Step S13: Fusing the semantic feature information with the spatial feature information, and judging whether the human target in the to-be-identified picture is in a sleeping state based on the fused feature information.

[0056] In this embodiment, the extracted semantic feature information is fused with the spatial feature information, and based on the fused information, it is judged whether the human target in the picture to be identified is in a sleeping state, so that the information in the picture to be identified can be fully utilized to judge the state of the human target.

[0057] It can be seen that in this embodiment, when performing sleeping post recognition, the head and shoulder area map corresponding to the human target and the corresponding target Gaussian heat map are obtained, and the head and shoulder area map and the target Gaussian heat map are respectively extracted and then feature fused, and the result after feature fusion is used to judge whether the corresponding human target is sleeping on the post. Therefore, in this application, when performing sleeping post recognition, it is only necessary to judge by the head and shoulder area features to avoid the problem of poor sleeping post recognition effect caused by failure to detect some key points when the target is sideways or severely blocked; on the other hand, by extracting the semantic features of the head and shoulder area map and the spatial features of the target Gaussian heat map and fusion of the features, various information in the image is fully utilized, thereby improving the accuracy of sleeping post recognition.

[0058] See also Figure 3 As shown, the embodiment of the present invention discloses a specific method for identifying sleeping guards. Compared with the previous embodiment, this embodiment further illustrates and optimizes the technical solution.

[0059] Step S21: input the image to be identified into the trained key point detection model to obtain the human key points of the image to be identified, and construct an original Gaussian heat map corresponding to the image to be identified based on the human key points of the image to be identified.

[0060] In this embodiment, the picture to be identified is input into the trained key point detection model to obtain the human key points of the picture to be identified, and the original Gaussian heat map corresponding to the picture to be identified is constructed based on the human key points of the picture to be identified, so as to subsequently obtain the target Gaussian heat map corresponding to the static human target from the original Gaussian heat map.

[0061] Step S22: obtaining a target picture group including the picture to be identified and several pictures adjacent to the picture to be identified, and determining whether the picture to be identified contains a static human target by detecting the position changes of key points between different adjacent pictures in the target picture group.

[0062] In this embodiment, the target picture group including the picture to be identified and several pictures adjacent to the picture to be identified refers to a target picture group consisting of multiple frames of pictures within a preset range before and after the picture to be identified. It is understandable that the number of pictures in the target picture group can be obtained by a preset interface and can be flexibly changed according to user needs, so as to perform key point detection on the above target picture group and then determine whether the picture to be identified contains a static human target.

[0063] Step S23: When the image to be identified contains a static human target, a head and shoulder area map corresponding to the static human target in the image to be identified is obtained.

[0064] Step S24: determining the Gaussian hot spots corresponding to the head-shoulder area map from the original Gaussian hot spot map to obtain a target Gaussian hot spot map corresponding to the head-shoulder area map.

[0065] In this embodiment, when the image to be identified contains a static human target, a head-shoulder area map corresponding to the static human target in the image to be identified is obtained, and a target Gaussian heat map corresponding to the head-shoulder area map is determined to facilitate subsequent extraction and feature fusion of spatial features and semantic features.

[0066] Step S25: extracting the semantic feature information of the head and shoulder area map and the spatial feature information of the target Gaussian heat map.

[0067] Step S26: Fusing the semantic feature information with the spatial feature information, and judging whether the human target in the to-be-identified picture is in a sleeping state based on the fused feature information.

[0068] In this embodiment, the semantic feature information of the head and shoulder area map and the spatial position relationship between different human key points corresponding to the target Gaussian heat map are extracted through the trained state classification model, and the semantic feature information and the spatial position information are feature fused. Then, the state of the human target in the image to be identified is classified using the fused feature information to obtain the corresponding human state category; the human state category includes a category corresponding to the sleeping state and a category corresponding to the non-sleeping state.

[0069] It can be seen that in this embodiment, key point detection is performed on a target picture group including the picture to be identified and several pictures adjacent to the picture to be identified, and the state of the human target is judged by the position change between key points. When the human target is determined to be dynamic, it is filtered; when the human target is static, it is input into the state classification model for sleeping position discrimination. Through the above technical solution, dynamic human targets can be screened out, avoiding unnecessary calculations caused by continuing to discriminate the sleeping position for dynamic human targets that are definitely not in the sleeping position state, and avoiding the problem of dynamic human targets being misjudged as sleeping position because they are in a sleeping position at a certain moment.

[0070] See also Figure 4 As shown, the embodiment of the present invention discloses a specific method for identifying sleeping guards. Compared with the previous embodiment, this embodiment further illustrates and optimizes the technical solution.

[0071] Step S31: input the picture to be identified into the trained human posture estimation model to perform human posture estimation on the picture to be identified through the human posture estimation model, obtain the human body key points detected in the human posture estimation process, and construct the original Gaussian heat map corresponding to the picture to be identified based on the human body key points of the picture to be identified; the human posture estimation model is constructed based on the Shuffle network and the HRNet network.

[0072] In this embodiment, a human body posture estimation model is used to obtain the human body key point detection of the image to be identified. Among them, the above-mentioned human body posture estimation model is based on Lite_HRNet as the basic network, Lite_HRNet is a human body posture estimation model built based on the Shuffle network and the HRNet network, and the above-mentioned human body posture estimation model has a strong representation ability and is more sensitive to position information. Compared with the mainstream lightweight network Lite_HRNet, it has lower computational complexity and higher performance. The L2 loss function is used in the training process to ensure that the network can converge. The key point confidence map output by the network is S, and the affinity vector is L. The loss function of each stage is calculated as follows:

[0073]

[0074]

[0075] In the formula, J represents the key point, C represents the limb, t represents the stage of progress, and P represents the position information of the key point. represents the key point confidence prediction value, represents the true value of the key point confidence, represents the predicted value of key point affinity, Represents the true value of the key point affinity, and ||·||2 represents the 2-norm. The human target key points in the image to be detected are located by the above-mentioned human posture estimation model, and the original Gaussian heat map corresponding to the image to be identified is constructed based on the human key points of the image to be identified, so as to determine the target Gaussian heat map through the original Gaussian heat map.

[0076] Step S32: obtaining a target picture group including the picture to be identified and several pictures adjacent to the picture to be identified, and determining whether the picture to be identified contains a static human target by detecting changes in key point positions between different adjacent pictures in the target picture group.

[0077] In this embodiment, the human body key points are identified for a target picture group including the picture to be identified and several pictures adjacent to the picture to be identified by a human body posture estimation model, and the key points are connected in the order of joints to obtain the skeleton of the human body. The human body posture estimation model is used in this embodiment to filter dynamic targets mainly because other methods, such as tracking and detection, are not effective for targets with small movement amplitudes, and the human body posture estimation model can well locate the position of each key point, facilitate accurate calculation of the change in the position of the key point between the previous and next frame pictures, thereby filtering out dynamic targets with small movement amplitudes, and greatly reducing the false alarm rate.

[0078] In this embodiment, the dynamic target filtering schematic diagram is as follows: Figure 5 As shown, the above target picture groups are input into the trained human posture estimation model to perform dynamic target filtering to obtain the human key points in each frame of the target picture group, and the human key points of the previous and next frame images are matched, and the change range of the human key point position is determined by calculating the mean square error value of the human key point position between the previous and next frames. The mean square error value is calculated as follows:

[0079]

[0080] In the formula, y i Indicates the location information of the key points located. Indicates the position information of the corresponding key points located in the previous frame, and n indicates the number of key points located. And by judging the size relationship between the mean square error value of the human key point position between the previous and next frames and the preset mean square error threshold, it is judged whether the human target corresponding to the above human key point position is in a static state. When the above mean square error value is greater than the preset mean square error threshold, the corresponding human target is judged to be a dynamic target; when the mean square error value is less than or equal to the preset mean square error threshold, the corresponding human target is judged to be a static human target. Furthermore, when performing human key point detection and state recognition on all human targets in the picture to be identified, the human target is assigned a unique identity information identifier and marked as the corresponding state.

[0081] Step S23: When the image to be identified contains a static human target, a head and shoulder area map corresponding to the static human target in the image to be identified is obtained; the human head and shoulder detection model is a model constructed based on the YOLO v5 network, and the Neck in the YOLO v5 network adopts a BiFPN structure.

[0082] In this embodiment, the image to be identified is input into the trained human head and shoulder detection model to obtain the head and shoulder area map corresponding to the human target in the image to be identified; the human head and shoulder detection model is a model constructed based on the YOLO v5 network, and the above-mentioned YOLO v5 network is mainly composed of four parts: input end, Backbone, Neck, and Predicition. The data set will be preprocessed at the network input end. The preprocessing used in the present invention mainly includes Mosaic data enhancement, adaptive image scaling, etc. In addition, the Backbone part of the YOLO v5 network uses CSP combined with Focus as the basic network, and Neck adopts the BiFPN structure. Different from the original FPN and PAN structures of YOLO v5, BiFPN is a weighted bidirectional feature pyramid network that can better fuse multi-scale information by introducing weights to weigh information of different scales. The loss function adopted is GIOU loss:

[0083]

[0084]

[0085] In the formula, A represents the real head and shoulder detection box, B represents the predicted head and shoulder detection box, and C represents the minimum rectangle containing A and B. The schematic diagram of head and shoulder area detection is shown in Figure 6 As shown, after the trained human head and shoulder detection model is used to identify the picture to be identified, the head and shoulder detection frames of all human targets in the picture to be identified are obtained, the identified human target head and shoulder detection frames are matched with the human key points in the human key point recognition, and the head and shoulder detection frames corresponding to the human targets marked as dynamic in the human key point detection are filtered out to obtain the head and shoulder detection frames corresponding to the static human targets, and a head and shoulder posture map is generated according to the head and shoulder detection frames corresponding to the static human targets, and the target Gaussian heat map corresponding to the head and shoulder posture map of the static human targets is intercepted from the original Gaussian heat map, and then the human target in the current frame of the picture to be identified is matched with the human target in the previous frame of the picture to be identified. The matching principle is to determine whether the two detection frames match by calculating the IoU (i.e., Intersection over Union) size of the detection frames in the previous and next frames and the preset IoU threshold. After the matching is completed, the state of the corresponding human target within a period of time is saved.

[0086] Step S34: determining the Gaussian hot spots corresponding to the head-shoulder area map from the original Gaussian hot spot map to obtain a target Gaussian hot spot map corresponding to the head-shoulder area map.

[0087] In this embodiment, the target Gaussian heat map is determined from the original Gaussian heat map according to the head and shoulder area map identified by the human head and shoulder detection model. The area relationship between the head and shoulder area map and the target Gaussian heat map is as follows: Figure 7 As shown, the head and shoulders area map has the same area boundary as the target Gaussian heat map, the format of the head and shoulders area map includes but is not limited to RGB format, and the target Gaussian heat map is determined by the boundary position of the head and shoulders area map.

[0088] Step S35: extract the semantic feature information of the head-shoulder area map and the spatial position relationship between different human key points corresponding to the target Gaussian heat map through the trained state classification model, and fuse the semantic feature information and the spatial position information, and then use the fused feature information to classify the state of the human target in the image to be identified to obtain the corresponding human state category; the human state category includes a category corresponding to the sleeping state and a category corresponding to the non-sleeping state.

[0089] In this embodiment, the sleep post process judgment diagram is as follows Figure 8 As shown, through the trained state classification model, the semantic feature information of the head and shoulder area map and the spatial position relationship between different human body key points corresponding to the target Gaussian heat map are extracted, and the semantic feature information and the spatial position information are feature fused, and then the sleeping state is judged. This can increase the spatial information of the head and shoulder part without losing the semantic information in the original head and shoulder area image, thereby improving the accuracy of sleeping state recognition.

[0090] In this embodiment, the GhostNet network is used as the basic network to construct the state classification model. The Ghost module used in the network can generate more feature maps through simple operations, fully reveal the internal feature information, and reduce the amount of calculation and improve the calculation speed under the same accuracy. In the training process of the state classification model, the ArcFace function is used as the loss function, and the calculation method is as follows:

[0091]

[0092] In the formula, n represents the number of samples, s and m are the hyperparameters in the training network, and y iis the label of the sample, and θ represents the angle of the feature vector. The above loss function can ensure that the inter-class distance is large enough and the intra-class distance is small enough to improve the robustness of the model. The GhostNet network in the state classification model is used to extract the semantic feature information of the head and shoulder area map and the spatial position relationship between different human key points corresponding to the target Gaussian heat map to obtain semantic feature information and spatial feature information, and the fused feature information is used to classify the state of the human target in the image to be identified to obtain the corresponding human state category. If the state classification model determines that the human target is in a sleeping state, 1 is cached in the buffer; if the state classification model determines that the human target is not in a sleeping state, 0 is cached in the buffer. When the number of 1s in the buffer exceeds the preset threshold, an alarm is triggered.

[0093] In this embodiment, the sleeping post identification schematic diagram is as follows Fig. 9 As shown, firstly, the image to be identified is input into the human posture recognition model, and the human key point detection and dynamic human target filtering are performed, and the original Gaussian heat map is generated according to the obtained human key point position information. Then, the identified image is input into the human head and shoulder detection model to determine the head and shoulder area map, and the target Gaussian heat map is determined according to the determined head and shoulder area map. The target Gaussian heat map and the head and shoulder area map are respectively input into the state classification model for spatial feature extraction and semantic feature extraction, and feature fusion is performed to determine whether the corresponding human target is in a sleeping state.

[0094] It can be seen that in this embodiment, the human posture estimation model with Lite_HRNet as the basic network is used for human key point recognition and dynamic human target filtering, which can accurately locate the position of human key points and facilitate the accurate calculation of the changes in the positions of the front and back key points, thereby filtering dynamic targets with smaller point motion amplitudes, greatly reducing the false alarm rate.

[0095] See also Fig.10 As shown, the embodiment of the present application discloses a sleeping post identification device, comprising:

[0096] A head-shoulder region determination module 11 is used to obtain a head-shoulder region map corresponding to a human target in a to-be-identified image, and determine a target Gaussian heat map corresponding to the head-shoulder region map;

[0097] A feature information extraction module 12, used to extract the semantic feature information of the head and shoulder area map and the spatial feature information of the target Gaussian heat map;

[0098] The state judgment module 13 is used to perform feature fusion on the semantic feature information and the spatial feature information, and judge whether the human target in the to-be-identified picture is in a sleeping state based on the fused feature information.

[0099] It can be seen that in this embodiment, when performing sleeping post recognition, the head and shoulder area map corresponding to the human target and the corresponding target Gaussian heat map are obtained, and the head and shoulder area map and the target Gaussian heat map are respectively extracted and then feature fused, and the result after feature fusion is used to judge whether the corresponding human target is sleeping on the post. Therefore, in this application, when performing sleeping post recognition, it is only necessary to judge by the head and shoulder area features to avoid the problem of poor sleeping post recognition effect caused by failure to detect some key points when the target is sideways or severely blocked; on the other hand, by extracting the semantic features of the head and shoulder area map and the spatial features of the target Gaussian heat map and fusion of the features, various information in the image is fully utilized, thereby improving the accuracy of sleeping post recognition.

[0100] In some specific embodiments, the head and shoulder region determination module 11 specifically includes:

[0101] A key point detection unit, used for inputting the image to be identified into a trained key point detection model to obtain the human key points of the image to be identified, and constructing an original Gaussian heat map corresponding to the image to be identified based on the human key points of the image to be identified;

[0102] The target Gaussian heat map determining unit determines the Gaussian heat map corresponding to the head-shoulder area map from the original Gaussian heat map to obtain a target Gaussian heat map corresponding to the head-shoulder area map.

[0103] In some specific embodiments, the sleeping post identification device further includes:

[0104] A static human target judging module is used to obtain a target picture group including the picture to be identified and a plurality of pictures adjacent to the picture to be identified, and to determine whether the picture to be identified contains a static human target by detecting the position change of key points between different adjacent pictures in the target picture group;

[0105] The static head-shoulder region map determining module is used to obtain the head-shoulder region map corresponding to the static human target in the image to be identified when the image to be identified contains a static human target.

[0106] In some specific embodiments, the sleeping post identification device further includes:

[0107] The posture model building module is used to build a human posture estimation model based on the Shuffle network and the HRNet network.

[0108] In some specific embodiments, the sleeping post identification device further includes:

[0109] The classification model building module is used to construct the state classification model using the GhostNet network as the basic network and the ArcFace function as the loss function.

[0110] Fig.11 The electronic device 20 provided in an embodiment of the present application is shown. The electronic device 20 may further include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the sleeping post identification method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may be specifically an electronic computer.

[0111] In this embodiment, the power supply 23 is used to provide voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0112] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0113] The operating system 221 is used to manage and control the hardware devices on the electronic device 20, and the computer program 222 can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the sleeper identification method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0114] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned sleeping post identification method is implemented. The specific steps of the method can be referred to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0115] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0116] The above is a detailed introduction to a sleeping post identification method, device, equipment and medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A sleeping post identification method, characterized in that: include: Obtain a head-shoulder region map corresponding to a human target in a to-be-recognized image, and determine a target Gaussian heat map corresponding to the head-shoulder region map; Extracting semantic feature information of the head and shoulder area map and spatial feature information of the target Gaussian heat map; The semantic feature information is fused with the spatial feature information, and based on the fused feature information, it is determined whether the human target in the to-be-identified picture is in a sleeping state; Extracting spatial feature information of the target Gaussian heat map, including: extracting spatial features from the target Gaussian heat map, first calculating the gradient value of each Gaussian hot spot in the target Gaussian heat map by back propagation, and taking an average value, performing spatial extraction on the average value to obtain a weight value of the spatial information feature map and multiplying the weight value with the feature value of each point of the spatial information feature map, adding and averaging the feature maps of the same layer to obtain a single feature map, and performing maximum and minimum normalization to obtain the spatial feature information of the target Gaussian heat map.

2. The sleeping post identification method according to claim 1, characterized in that: The step of obtaining a head and shoulder area map corresponding to a human target in a to-be-recognized image includes: The image to be identified is input into the trained human head and shoulder detection model to obtain a head and shoulder area map corresponding to the human target in the image to be identified; the human head and shoulder detection model is a model built based on the YOLO v5 network, and the Neck in the YOLO v5 network adopts a BiFPN structure.

3. The sleeping post identification method according to claim 1, characterized in that: The determining of a target Gaussian heat map corresponding to the head and shoulders area map includes: Input the image to be identified into the trained key point detection model to obtain the human body key points of the image to be identified, and construct an original Gaussian heat map corresponding to the image to be identified based on the human body key points of the image to be identified; A Gaussian hot spot corresponding to the head-shoulder area map is determined from the original Gaussian hot spot map to obtain a target Gaussian hot spot map corresponding to the head-shoulder area map.

4. The sleeping post identification method according to claim 3, characterized in that: Also includes: Acquire a target picture group including the picture to be identified and several pictures adjacent to the picture to be identified, and determine whether the picture to be identified includes a static human target by detecting changes in key point positions between different adjacent pictures in the target picture group; Accordingly, the step of obtaining a head and shoulder area map corresponding to a human target in a to-be-recognized image includes: When the image to be identified contains a static human target, a head and shoulder area map corresponding to the static human target in the image to be identified is obtained.

5. The sleeping post identification method according to claim 4, characterized in that: Also includes: Construct a human posture estimation model based on the Shuffle network and the HRNet network; Accordingly, the step of inputting the image to be identified into the trained key point detection model to obtain the human key points of the image to be identified includes: The image to be identified is input into the trained human posture estimation model, so as to perform human posture estimation on the image to be identified through the human posture estimation model, and obtain the key points of the human body detected in the human posture estimation process.

6. The sleeping post identification method according to any one of claims 1 to 5, characterized in that: The extracting of the semantic feature information of the head-shoulder area map and the spatial feature information of the target Gaussian heat map, fusing the semantic feature information with the spatial feature information, and judging whether the human target in the to-be-identified picture is in a sleeping state based on the fused feature information includes: Through the trained state classification model, the semantic feature information of the head and shoulder area map and the spatial position relationship between different human key points corresponding to the target Gaussian heat map are extracted, and the semantic feature information and the spatial position relationship are feature fused. Then, the state of the human target in the image to be identified is classified using the fused feature information to obtain the corresponding human state category; the human state category includes a category corresponding to the sleeping state and a category corresponding to the non-sleeping state.

7. The sleeping post identification method according to claim 6, characterized in that: Also includes: Using the GhostNet network as the basic network and the ArcFace function as the loss function, constructing the state classification model; Among them, the GhostNet network in the state classification model is used to extract the semantic feature information of the head and shoulder area map and the spatial position relationship between different human body key points corresponding to the target Gaussian heat map.

8. A sleeping post identification device, characterized in that: include: A head-shoulder region determination module is used to obtain a head-shoulder region map corresponding to a human target in a to-be-identified image, and determine a target Gaussian heat map corresponding to the head-shoulder region map; A feature information extraction module, used to extract the semantic feature information of the head and shoulder area map and the spatial feature information of the target Gaussian heat map; A state judgment module, used for performing feature fusion of the semantic feature information and the spatial feature information, and judging whether the human target in the to-be-identified picture is in a sleeping state based on the fused feature information; The feature information extraction module is specifically used to extract spatial features from the target Gaussian heat map. First, the gradient value of each Gaussian hot spot in the target Gaussian heat map is calculated by back propagation, and the average value is taken. The weight value of the spatial information feature map obtained after spatial extraction of the average value is multiplied by the feature value of each point in the spatial information feature map. The feature maps of the same layer are added and averaged to obtain a single feature map for maximum and minimum normalization to obtain the spatial feature information of the target Gaussian heat map.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the steps of the sleeping post identification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, the steps of the sleeping post identification method as described in any one of claims 1 to 7 are implemented.

Images