A sleep state recognition method, device, apparatus and storage medium

Abstract

The application discloses a sleep state recognition method and device, equipment and a storage medium, relates to the technical field of image monitoring and detection, and comprises the following steps: inputting an original image of a current frame collected by a camera into a target detection model constructed in advance to obtain a human upper body detection frame and a human upper body skeleton key point coordinate sequence; inputting a human upper body region image obtained by cutting into a residual network to obtain a shooting angle of the camera relative to a personnel target; if the shooting angle is front shooting, side shooting or top shooting, inputting the personnel target into a first sleep state classification model to obtain a first classification result of whether the personnel target is in a sleep state; and if the shooting angle is back shooting, comprehensively judging whether the personnel target in the current frame is in a sleep state by using the human upper body skeleton key point coordinate sequence and image features output by the residual network. The application adopts different recognition strategies for personnel targets with different shooting angles, and can improve the recognition accuracy under back shooting.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image monitoring detection, and in particular to a sleep post recognition method, device and equipment and storage medium. BACKGROUND

[0002] At present, in various industries, safety production is an important factor for enterprises and individuals, and is of great significance to the healthy development of national economy and people's livelihood. In the fields of industrial manufacturing, petroleum chemical industry and the like, production safety is more important. Enterprises often arrange post personnel to view abnormal events in each monitoring point through monitoring pictures in real time, so as to discover safety hazards early and deal with safety events early.

[0003] However, how to prevent the post personnel from sleeping during working hours to cause abnormal events to be not discovered in time and cause major safety accidents has been a problem for enterprise managers to think about and solve. At present, there are many detection algorithms for sleep post behavior of post personnel on the market, some of which use sensors to obtain offline data for sleep post behavior monitoring, and some of which are based on monitoring video to recognize and classify the sleep post behavior of the post personnel. Relatively speaking, the method of image analysis on the video picture of the post room has better recognition effect and higher accuracy, but this kind of method is usually some recognition technology based on deep learning to analyze the human posture, and in the indoor scene of the office, the recognition accuracy of the backshot target is often low, and false positives of non-sleep post of the backshot target are easy to occur. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a sleep post recognition method, device, equipment and storage medium, which can improve the recognition rate of sleep post violation behavior and ensure the recognition accuracy of sleep post violation behavior. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a sleep post recognition method, comprising:

[0006] inputting an original image of a current frame collected by a camera into a pre-constructed target detection model to obtain a human upper body bounding box and a human upper body skeleton key point coordinate sequence in the original image output by the target detection model;

[0007] cropping the original image according to the human upper body bounding box to obtain a human upper body region image, and inputting the human upper body region image into a residual network to obtain a shooting angle of the camera relative to a personnel target recognized and output by the residual network;

[0008] if the shooting angle is front shot, side shot or top shot, inputting the personnel target into a first sleep post classification model to obtain a first classification result of whether the personnel target in the current frame is in a sleep post state output by the first sleep post classification model;

[0009] If the shooting angle is back shooting, a skeletal point spatial relationship is calculated by using the human upper body skeletal key point coordinate sequence, and whether the personnel target in the current frame is in a sleep state is comprehensively judged by using the skeletal point spatial relationship and the image feature output by the residual network.

[0010] In a second aspect, the present application discloses a sleep state recognition device, comprising:

[0011] A first detection module is configured to input an original image of a current frame collected by a camera into a pre-constructed target detection model to obtain a human upper body detection frame and a human upper body skeletal key point coordinate sequence in the original image output by the target detection model;

[0012] A cropping module is configured to crop the original image according to the human upper body detection frame to obtain a human upper body region image;

[0013] A second detection module is configured to input the human upper body region image into a residual network to obtain a shooting angle of the camera relative to a personnel target output by the residual network;

[0014] A first sleep state recognition module is configured to, when the shooting angle is front shooting, side shooting or top shooting, input the personnel target into a first sleep state classification model to obtain a first classification result of whether the personnel target in the current frame is in a sleep state output by the first sleep state classification model;

[0015] A second sleep state recognition module is configured to, when the shooting angle is back shooting, calculate a skeletal point spatial relationship by using the human upper body skeletal key point coordinate sequence, and comprehensively judge whether the personnel target in the current frame is in a sleep state by using the skeletal point spatial relationship and the image feature output by the residual network.

[0016] In a third aspect, the present application discloses an electronic device, comprising:

[0017] A memory is configured to save a computer program;

[0018] A processor is configured to execute the computer program to realize the steps of the sleep state recognition method disclosed above.

[0019] In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to realize the steps of the sleep state recognition method disclosed above.

[0020] It can be seen that the application provides a sleep state recognition method, which comprises the following steps: inputting an original image of a current frame collected by a camera into a pre-constructed target detection model to obtain a human upper body bounding box and a human upper body skeleton key point coordinate sequence output by the target detection model; cropping the original image according to the human upper body bounding box to obtain a human upper body region image, and inputting the human upper body region image into a residual network to obtain a shooting angle of the camera relative to a target person recognized and output by the residual network; if the shooting angle is front shooting, side shooting or top shooting, inputting the target person into a first sleep state classification model to obtain a first classification result of whether the target person in the current frame is in a sleep state output by the first sleep state classification model; if the shooting angle is back shooting, calculating a skeleton point spatial relationship by using the human upper body skeleton key point coordinate sequence, and comprehensively judging whether the target person in the current frame is in a sleep state by using the skeleton point spatial relationship and an image feature output by the residual network. It can be seen that the application identifies the shooting angle based on the human upper body region image, and then adopts different recognition strategies for the target person in different shooting angles, so as to fully utilize the human upper body skeleton key point coordinate sequence and the human upper body image feature information to recognize the sleep state of the target person, and improve the recognition accuracy of the sleep state of the target person in the back shooting of the camera. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on the provided drawings.

[0022] Figure 1 A sleep state recognition method flow chart disclosed by the present application;

[0023] Figure 2 A specific sleep state recognition method flow chart disclosed by the present application;

[0024] Figure 3 A human body skeleton key point schematic diagram disclosed by the present application;

[0025] Figure 4 A right-angle coordinate system schematic diagram with point A1 as the coordinate origin disclosed by the present application;

[0026] Figure 5 A right-angle coordinate system schematic diagram with point A2 as the coordinate origin disclosed by the present application;

[0027] Figure 6A point A distance relationship calculation schematic diagram disclosed by the present application;

[0028] Figure 7 A point A angle relationship calculation schematic diagram disclosed by the present application;

[0029] Figure 8 A voting mechanism flow chart based on a time sequence frame disclosed by the present application;

[0030] Figure 9 A time sequence frame state switching proportion calculation schematic diagram disclosed by the present application;

[0031] Figure 10 A time sequence frame recognition result judgment method flow chart disclosed by the present application;

[0032] Figure 11 An initial video frame sequence recognition result schematic diagram disclosed by the present application;

[0033] Figure 12 A modified video frame sequence recognition result schematic diagram disclosed by the present application;

[0034] Figure 13 A sleep post recognition device structure schematic diagram disclosed by the present application;

[0035] Figure 14 An electronic device structure diagram disclosed by the present application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0037] Currently, how to prevent the value post personnel from sleeping during working hours to cause abnormal events to be not discovered in time and cause major safety accidents has always been a problem for enterprise managers to think about and solve. Currently, there are some methods of monitoring the sleep post behavior of value post personnel by using sensors to obtain offline data, and some methods of identifying and classifying the sleep post behavior of value post personnel based on monitoring video. Relatively speaking, the method of image analysis on the video picture of the value post room has better recognition effect and higher accuracy, but this kind of method is usually some recognition technology based on deep learning technology to analyze the human posture. In the case of back shooting of personnel target in indoor scenes such as office, the recognition accuracy is often low, and false positives of non-sleep post of back shooting target are easy to occur. Therefore, the present application provides a sleep post recognition scheme, which can improve the recognition rate of sleep post violation behavior and ensure the recognition accuracy of sleep post violation behavior.

[0038] The embodiment of the present application discloses a sleep state recognition method, referring to Figure 1 The method comprises the following steps:

[0039] Step S11: input the original image of the current frame collected by the camera into the pre-constructed target detection model to obtain the upper body detection frame and the upper body skeleton key point coordinate sequence of the original image output by the target detection model.

[0040] It can be understood that the region detection and skeleton key point positioning of the upper body are realized by the pre-constructed target detection model, and the upper body detection frame and the upper body skeleton key point coordinates are obtained. Moreover, the upper body detection algorithm in the pre-constructed target detection model can include but is not limited to the RCNN (Region-CNN) series, the SSD (Solid State Drive) series or the YOLO series algorithm. For example, the region detection and skeleton key point positioning of the upper body in the original image are realized by the target detection model pre-constructed based on the YOLO V5 algorithm, and the upper body detection frame and the upper body skeleton key point coordinates are obtained.

[0041] Step S12: crop the original image according to the upper body detection frame to obtain an upper body region image, and input the upper body region image into a residual network to obtain the shooting angle of the camera relative to the personnel target recognized and output by the residual network.

[0042] In the embodiment, after obtaining the upper body detection frame of the original image output by the pre-constructed target detection model, the original image is cropped according to the upper body detection frame to obtain an upper body region image, and then the upper body region image is taken as the input image of the camera shooting angle recognition, that is, the upper body region image is input into the residual network to obtain the shooting angle of the camera relative to the personnel target recognized and output by the residual network. It can be understood that the residual network is used to realize the recognition of the shooting angle of the camera relative to the personnel target, and the residual network can be a ResNet18 residual network. For example, the ResNet18 residual network is used to realize the recognition of the shooting angle of the camera relative to the personnel target, and four shooting angle categories including front shooting, side shooting, top shooting and back shooting are output.

[0043] Step S13: if the shooting angle is front shooting, side shooting or top shooting, input the personnel target into a first sleep state classification model to obtain the first classification result of whether the personnel target in the current frame is in a sleep state output by the first sleep state classification model.

[0044] It can be understood that, for the camera front, side and overhead angle of the personnel target, the application completes the classification and judgment of whether the personnel target is in the sleep state through a simple first sleep classification model. Moreover, the first sleep classification model is a model based on CNN (Convolutional Neural Network), a full connection layer and a softmax layer, wherein the CNN is used to extract image features.

[0045] Step S14: if the shooting angle is back, the skeletal point space relationship is calculated by using the human upper body skeletal key point coordinate sequence, and whether the personnel target in the current frame is in the sleep state is comprehensively judged by using the skeletal point space relationship and the image features output by the residual network.

[0046] It can be understood that, because of the influence of the camera shooting angle and the prone sleep posture of the human body, the prone and upright postures of the personnel target under the back shooting angle are difficult to accurately identify by using a simple sleep classification model. Therefore, for the personnel target state under the camera back shooting angle which is difficult to distinguish, the application not only uses the method of fusing and re-identifying the skeletal key point space information and the image deep feature information, but also uses the skeletal key point space information for logical judgment, and finally the discrimination results of the two are summarized by using confidence weighting, so as to obtain the comprehensive discrimination result of whether the personnel target in the current frame is in the sleep state.

[0047] In the embodiment, after the comprehensive discrimination result of whether the personnel target in the current frame is in the sleep state is determined, it can further include: judging whether the comprehensive discrimination result meets the corresponding alarm condition, if the comprehensive discrimination result meets the alarm condition, it is determined that the personnel target in the original image of the current frame is in the sleep state, and the corresponding sleep violation behavior alarm is triggered, if the comprehensive discrimination result does not meet the alarm condition, it is determined that the personnel target in the original image of the current frame is in the non-sleep state, and no alarm is needed.

[0048] It can be seen that, in the embodiment of the application, the shooting angle is identified based on the human upper body region image, and then different identification strategies are used for the target personnel under different shooting angles, so that the human upper body skeletal key point coordinate sequence and the human upper body image feature information can be fully utilized for the identification of the sleep state of the personnel target, and the identification accuracy of the sleep state of the personnel target under the camera back shooting can be improved.

[0049] Referring to Figure 2 The embodiment of the application discloses a specific sleep identification method, and compared with the previous embodiment, the technical solution is further described and optimized.

[0050] Step S21: input the original image of the current frame collected by the camera into the pre-constructed target detection model to obtain the upper body detection frame of the human body and the sequence of upper body skeleton key point coordinates output by the target detection model.

[0051] Step S22: crop the original image according to the upper body detection frame to obtain an upper body region image, and input the upper body region image into the residual network to obtain the shooting angle of the camera relative to the personnel target recognized and output by the residual network.

[0052] Step S23: if the shooting angle is front, side or top, input the personnel target into the first sleep state classification model to obtain the first classification result of whether the personnel target in the current frame is in a sleep state output by the first sleep state classification model.

[0053] Step S24: if the shooting angle is back, calculate the spatial relationship of the skeleton points by using the sequence of upper body skeleton key point coordinates, and determine the sleep score of the personnel target in the current frame in a sleep state and the still score of the personnel target in the current frame in a still state based on the spatial relationship of the skeleton points.

[0054] In the embodiment, before the spatial relationship of the skeleton points is calculated by using the sequence of upper body skeleton key point coordinates, the confidence of each skeleton key point is determined based on the first confidence score corresponding to each skeleton key point output by the target detection model; if the confidence is less than a pre-set confidence filtering threshold, it is determined that the skeleton key point is an abnormal point, and the predicted coordinates of the abnormal point determined based on the prediction rule of the skeleton key point are used to replace the abnormal coordinates of the abnormal point; if the confidence is not less than the pre-set confidence filtering threshold, it is determined that the skeleton key point is a non-abnormal point. It can be understood that the process of changing the target from standing to lying prone state is easy to cause inaccurate positioning of part of the skeleton points of the human body, that is, the sequence of upper body skeleton key point coordinates output by the pre-constructed target detection model may have abnormal skeleton key point coordinates, therefore, these abnormal points need to be identified and corrected. When the pre-constructed target detection model outputs the position coordinates of each skeleton key point, it also gives the first confidence score corresponding to the skeleton key point, the confidence of the corresponding skeleton key point is determined based on the first confidence score corresponding to the skeleton key point coordinates output by the target detection model, whether the skeleton key point is an abnormal point is determined based on the confidence, and then the predicted coordinates of the abnormal point are used to correct the abnormal coordinates of the abnormal point.

[0055] It should be noted that, since the sleep behavior is a time-sequential and continuous process, the movement trend of the same bone point has certain angle and distance rules in the process of moving from the upright state to the prone state, and the movement trend changes little between adjacent frames. As shown in Figure 3 , five key points of the human head (point A), neck (point B), back (point C), left shoulder (point D), and right shoulder (point E) are selected as analysis objects, and the key point coordinates of the indoor back-shooting personnel database are analyzed by big data to obtain the following conclusions: first, the change amplitude of the head (point A) is the largest in the change process of the human body moving from the upright state to the prone state, and the floating amplitude of the remaining points is small; second, the ratio of the distance between points BC to the distance between points AB gradually increases in the change process of the human body moving from the upright state to the prone state, and the more obvious the prone posture, the greater the inter-frame ratio change rate; third, the closer the camera is to the back-shooting, the smaller the starting value of the distance ratio change interval in the change process of moving from the upright state to the prone state, for example, the change interval of the back-shooting is [1.5, 2.0], and the change interval of the back-shooting is [1.0, 1.6]. Based on the above conclusions, taking the head (point A) as an example, assuming that the target personnel is in the upright state, the first frame A point, i.e., point A1, is at the top of the screen, and point A2 represents the position of the second frame A point, as shown in Figure 4 , a plane rectangular coordinate system is established with point A1 as the coordinate origin, and the position relationship between point A1 and point A2 is as shown in Figure 4 .

[0056] dis 12 represents the distance between point A1 and point A2, v 12 represents the speed of moving from point A1 to point A2, and a 12 represents the change angle when moving from point A1 to point A2.

[0057] When the time interval between the first frame and the second frame is t 12 , the distance between point A1 and point A2 is:

[0058] dis 12 = v 12 * t 12 .

[0059] Since the time interval of each frame is the same, i.e., t 12 = t 23 = t 34 = t... = t, the prediction rule of the bone key point can be:

[0060] dis i，i+1 = dis i-1，i *(v i-1，i / v i-2，i-1 ).

[0061] ai，i+1 =α i-1，i *(α i-1，i / α i-2，i-1 );

[0062] Therefore, based on the motion pattern of point A2 relative to point A1, we can predict the position of point A in the 3rd frame, that is, point A3, and point A3 continues the motion trend of point A2.

[0063] Since point A1 is the initial point, let v be... 01 ≈v 12 α 01 ≈α 12 Therefore, as Figure 5 As shown, a Cartesian coordinate system is established with point A2 as the origin. The angle between the distance between points A2 and A3 and the angle between them when point A2 moves to point A3 is as follows:

[0064] dis 23 =dis 12 *(v 12 / v 01 ), v 01 ≈v 12 ;

[0065] α 23 =α 12 *(α 12 / α 01 ), α 01 ≈α 12 ;

[0066] Therefore, the distance between points A2 and A3 is dis 23 equal to the distance dis between points A1 and A2 12 The angle α changes as point A2 moves to point A3. 23 The angle α is equal to the change in angle when point A1 moves to point A2. 12 ,Right now:

[0067] dis 23 =dis 12 ;

[0068] α 23 =α 12 ;

[0069] Similarly, the predicted position of point A in the fourth frame, i.e., the predicted position of A4, can be:

[0070] dis 34 =dis 23 *(v 23 / v 12 );

[0071] α 34 =α34 *(α 34 / α 12 )。

[0072] Specifically, the confidence of each of the skeletal keypoints can be determined based on the first confidence score corresponding to each of the skeletal keypoints output by the target detection model. For example, if the first confidence score corresponding to a skeletal keypoint output by the target detection model is not less than a first preset threshold, the first confidence score is determined as the confidence of the skeletal keypoint. If the first confidence score is greater than a second preset threshold and less than the first preset threshold, a probability that the skeletal keypoint is the non-anomalous point is determined, and the confidence of the skeletal keypoint is calculated according to the first confidence score and the probability. If the first confidence score is not greater than the second preset threshold, a position relationship score of the skeletal keypoint is determined, and the probability that the skeletal keypoint is the non-anomalous point is determined, and the confidence of the skeletal keypoint is calculated according to the first confidence score, the position relationship score and the probability.

[0073] Specifically, the probability that the skeletal keypoint is the non-anomalous point can be determined according to a preset keypoint prediction method based on a prediction confidence of the skeletal keypoint. For example, according to the preset keypoint prediction method, the accuracy of each of the skeletal keypoints output by the target detection model can be determined based on a prediction point coordinate. In the preset keypoint prediction method, the confidence of an i-th frame skeletal keypoint is defined as an accumulation of a plurality of frame prediction confidences, and the calculation formula of the confidence of the i-th frame skeletal keypoint is as follows:

[0074] P pre，i i-1 pre，i-1 +(1-β1)*P′ pre，i i-1 ;

[0075] wherein P pre，i i-1 represents the accumulation of the prediction confidences of the skeletal keypoints in the i-th frame, P′ pre，i i-1 represents the prediction confidence of the skeletal keypoint in the i-th frame, P pre，i-1 i-1 represents the accumulation of the prediction confidences of the skeletal keypoints in the i-1-th frame, and β1 represents a weight hyperparameter of the prediction confidences of the i-th frame and the i-1-th frame.

[0076] Therefore, the position prediction range of the skeletal keypoint in the next frame, i.e., the distance prediction range and the angle prediction range, can be determined based on the prediction confidence of the skeletal keypoint in the current frame.

[0077] Taking the head point A as an example, assuming that the prediction credibility of point A1 is P pre，1 , and the prediction credibility of point A2 is P pre，2 , then the distance prediction range of the coordinate position of point A in the third frame is:

[0078] [P pre，2 *dis 2，2′ , (2-P pre，2 )*dis 2，2′ ];

[0079] The angle prediction range of the coordinate position of point A in the third frame is:

[0080] [P pre，2 *α 2，2′ , (2-P pre，2 )*α 2，2′ ];

[0081] dis 2，2′ represents the coordinate distance between the predicted value and the real value of point A in the second frame, and α 2，2′ represents the angle distance between the predicted value and the real value of point A in the second frame.

[0082] Based on the above position prediction range, the probability that the real point of point A in the third frame is a non-abnormal point is calculated, and the related calculation formula is as follows:

[0083]

[0084] dis 3，3′ represents the coordinate distance between the predicted value and the real value of point A in the third frame, and α 3，3′ represents the angle distance between the predicted value and the real value of point A in the third frame.

[0085] Specifically, the determining the position relationship score of the skeletal key point can include: determining the position relationship between each skeletal key point of the personnel target; the position relationship includes a distance relationship and an angle relationship; determining whether the skeletal key point is the abnormal point based on the position relationship to obtain a corresponding first judgment result; determining the position relationship score of the skeletal key point based on the first judgment result; the position relationship score includes a distance relationship score and an angle relationship score. It can be understood that in the same video picture, there is a certain position relationship between each skeletal key point of the same target, mainly including a distance relationship and an angle relationship.

[0086] For example, according to the distance relationship, it is judged whether the skeletal key point is an abnormal point, such as Figure 6As shown, taking the head (point A) with the largest variation amplitude as an example, a plane rectangular coordinate system is established with the left shoulder (point D) and the right shoulder (point E) of the human body as the horizontal axis, and point B is selected to set the vertical axis of the coordinate system, and the positional relationship between point A and the other points is calculated. When the personnel target is in an upright state, based on the different camera shooting angles, the ratio of the distance from the head point A to the neck point B to the distance from the neck point B to the waist point C, i.e. the interval of dis BC / dis AB , is [1.0, 2.0], wherein dis AB represents the distance from the head point A to the neck point B, dis BC represents the distance from the neck point B to the waist point C, and dis BC is the length of the distance from the head point A to the waist point C. A circle is drawn with 0.5 times and 1.0 times the length of dis BC as the radius, and when the skeletal key point coordinate falls in the middle region between the two circles, it is determined that the skeletal key point is a non-abnormal point, such as point A1 and point A3 in the above Figure 8 , that is, point A1 and point A3 are non-abnormal points, so the distance relationship score of points A and B is:

[0087]

[0088] If the skeletal key point coordinate does not fall in the middle region between the two circles, it is determined that the skeletal key point is an abnormal point, such as point A4 in the above Figure 6 , that is, point A4 is an abnormal point, so the distance relationship score of points A and B is:

[0089]

[0090] , wherein R1 represents a radius of 1.0 times the length of dis BC , that is, R1 = 1*dis BC , and R2 represents a radius of 0.5 times the length of dis BC , that is, R2 = 0.5*dis BC .

[0091] For example, according to the angle relationship, whether the skeletal key is an abnormal point is determined, as shown in Figure 7 , since the head position of a person in a normal lying state cannot be below the shoulder. Therefore, the head point A can only be in the first quadrant and the second quadrant of the coordinate system, and the line AB is at a certain angle with the coordinate system, and an angle threshold is set as a, and two angle threshold lines are drawn, and when the included angle θ of the line AB is less than the angle threshold a, that is, when the point A falls below the angle line, it is determined that the point A is an abnormal point, such as point A2 and point A4, that is:

[0092]

[0093] Correspondingly, the angle relationship score corresponding to the included angle θ of the line AB is:

[0094]

[0095] If the included angle θ of the line AB is not less than the included angle threshold a, that is, the point A is not located in the position below the angle line, the point A is determined as the non-anomaly point, such as the points A1 and A3, and the angle relationship score corresponding to the included angle θ of the line AB is:

[0096]

[0097] The included angle threshold a represents an adjustable angle threshold, and the value range of a can be [30°, 70°].

[0098] For example, the first confidence score corresponding to the skeletal key point and the confidence of the skeletal key point are in the following relationship:

[0099]

[0100] P conf represents the first confidence score of the skeletal key point, P i，A represents the confidence of the i-th frame key point A, P pre，true represents the probability that the skeletal key point is determined as the non-anomaly point, P dis represents the distance relationship score, and P α represents the angle relationship score.

[0101] That is, when the first confidence score of the i-th frame key point A output by the target detection model is greater than 0.9, the confidence of the i-th frame key point A is:

[0102] P i，A = P conf ;

[0103] When the first confidence score of the i-th frame key point A output by the target detection model is greater than 0.7 and less than 0.9, the confidence of the i-th frame key point A is:

[0104] P i，A = P pre，ture · P conf ;

[0105] That is, when the first confidence score of the i-th frame key point A output by the target detection model is greater than 0.7 and less than 0.9, the position relationship between the predicted point and the real point is introduced for judgment.

[0106] When the first confidence score of the i-th frame key point A output by the target detection model is less than 0.7, the confidence of the i-th frame key point A is:

[0107] P i，A = P dis · Pα ·P pre，ture ·P conf ;

[0108] wherein, only P dis ·P α satisfy the filtering condition, the confidence of the key point A in the i-th frame is P i，A =P pre，true ·P conf , if P dis ·P α does not satisfy the filtering condition, then the confidence of the key point A in the i-th frame is P i，A =0.

[0109] Therefore, when the confidence of the key point A in the i-th frame is less than a preset confidence filtering threshold, i.e., P i，A <P th , it is determined that the key point A in the i-th frame is an abnormal point, and the abnormal point is repaired; if the confidence of the key point A in the i-th frame is not less than the preset confidence filtering threshold, it is determined that the key point A in the i-th frame is a non-abnormal point, wherein P th represents the confidence filtering threshold, and is set as 0.6 by default. For example, assuming that the key point A is determined to be an abnormal point in the fourth frame, i.e., P 4，A <P th，A , the key point A in the fourth frame can be repaired according to the predicted point A4' of the key point A in the fourth frame, wherein according to the prediction rule of the key point, it can be known that the predicted point of the key point A in the fourth frame has the following positional relationship:

[0110] dis 34 =dis 23 *(v 23 / v 12 );

[0111] α 34 =α 34 *(α 34 / α 12 );

[0112] And it is known that the distance calculation formula of the target key point in the i-th frame and the target key point in the i-1-th frame is:

[0113] dis i-1，i =v i-1，i *t i-1，i ;

[0114] t 12 =t 23 =t 34 =t...=t i-1，i ;

[0115] Therefore, the distance between the third frame A point and the fourth frame A point, i.e. the distance between point A3 and point A4, is:

[0116]

[0117] Since A1 has coordinates (x1, y1), A2 has coordinates (x2, y2), and A3 has coordinates (x3, y3), then:

[0118]

[0119]

[0120]

[0121] Substituting the coordinates, the distance between the first frame A point and the second frame A point, i.e. the distance between point A1 and point A2, dis 12 , and the distance between the second frame A point and the third frame A point, i.e. the distance between point A2 and point A3, dis 23 , can be calculated. According to the angle relationship, the change angle a when point A3 moves to point A4 can be determined. 34 Finally, the predicted coordinates of the predicted point A4' of point A in the fourth frame are confirmed, and the abnormal coordinates of point A, i.e. A4, in the fourth frame are replaced by the predicted coordinates of the predicted point A4'.

[0122] In this embodiment, after repairing the abnormal point, the skeletal point spatial relationship can be calculated using the human upper body skeletal key point coordinate sequence, and the sleep state score of the personnel target in the current frame and the still state score of the personnel target in the current frame can be determined based on the skeletal point spatial relationship. It can be understood that logical judgment using spatial information between skeletal key points can realize the recognition of the sleep behavior of a single frame target, i.e. determining the sleep state score of the personnel target in the current frame and the still state score of the personnel target in the current frame based on the skeletal point spatial relationship.

[0123] In this embodiment, the calculation of the spatial relationship of the skeletal points based on the sequence of the upper body skeletal key point coordinates of the human body can include: calculating a first distance between a head key point and a neck key point and a second distance between the neck key point and a waist key point based on the sequence of the upper body skeletal key point coordinates of the human body, and determining the displacement distance of each skeletal key point between adjacent two frames; accordingly, the determination of the sleep posture score of the personnel target in the current frame in the sleep posture state and the stillness score of the personnel target in the current frame in the stillness state based on the spatial relationship of the skeletal points can include: determining the ratio between the first distance and the second distance; determining a second judgment result corresponding to whether the ratio meets a preset condition, and determining the sleep posture score of the personnel target in the current frame in the sleep posture state based on the second judgment result; determining the stillness state score of each skeletal key point in the stillness state based on a first comparison result of the displacement distance of each skeletal key point between adjacent two frames and a preset displacement threshold, and calculating the stillness score of the personnel target in the current frame in the stillness state based on the stillness state score corresponding to each skeletal key point. For example, the ratio of the distance dis AB between the head point A and the neck point B to the distance dis BC between the neck point B and the waist point C, according to the analysis of the skeletal key point data in the sleep posture behavior database, when the personnel target is in the sleep posture, the ratio of the distance dis AB between the head point A and the neck point B to the distance dis BC between the neck point B and the waist point C will be in the interval of 1.8 to 2.0, that is , and the sleep posture score of the personnel target in the sleep posture state can be determined as P sleep，i = 1; if the ratio of the distance dis AB between the head point A and the neck point B to the distance dis BC between the neck point B and the waist point C is not in the interval of 1.8 to 2.0, the sleep posture score of the personnel target in the sleep posture state can be determined as P sleep，i = 0.

[0124] The stillness state score of each skeletal key point in the stillness state is determined based on the movement distance of the skeletal key point relative to the previous frame, for example, when the movement distance of the skeletal key point between adjacent two frames is greater than a threshold, it is determined that the skeletal key point is in a non-stillness state, and the related calculation process is as follows:

[0125] When the coordinates of each skeletal key point in the i-th frame are:

[0126] A i (x A，i , y A，i ), B i (xB，i , y B，i ), C i (x C，i , y C，i ), D i (x D，i , y D，i ), E i (x E，i , y E，i ), F i (x F，i , y F，i ), G i (x G，i , y G，i ), H i (x H，i , y H，i ), I i (x I，i , y I，i );

[0127] When the i-1th frame of each bone key point coordinates are in turn:

[0128] A i-1 (x A，i-1 , y A，i-1 ), B i-1 (x B，i-1 , y B，i-1 ), C i-1 (x C，i-1 , y C，i-1 ), D i-1 (x D，i-1 , y D，i-1 ), E i-1 (x E，i-1 , y E，i-1 );

[0129] F i-1 (x F，i-1 , y F，i-1 ), G i-1 (x G，i-1 , y G，i-1 ), H i-1 (x H，i-1 , y H，i-1 ), I i-1 (x I，i-1 , y I，i-1 );

[0130] Taking point A as an example, the displacement distance of point A in the i-th frame and point A in the i-1th frame is:

[0131]

[0132] And when the displacement distance of point A between two frames is greater than the preset displacement threshold, i.e. dis A > dis A，th , it is determined that the i-th frame point A is in a motion state, i.e. the static state score of the i-th frame point A in a static state is:

[0133] P staic，i，A = 0.

[0134] Based on this, the static state scores of the other 8 points in the static state are calculated, and finally the static score of the personnel target in the current frame in the static state is obtained through voting, i.e.:

[0135]

[0136] Step S25: The skeletal point spatial relationship and the image features output by the residual network are fused to obtain a corresponding fusion result.

[0137] In this embodiment, after calculating the skeletal point spatial relationship using the human upper body skeletal key point coordinate sequence, the skeletal point spatial relationship and the image features output by the residual network are fused to obtain a corresponding fusion result for subsequent single-frame target sleep behavior recognition classification using the fusion result.

[0138] It should be noted that the fusion method of image deep features and skeletal key point spatial information can include but is not limited to addition, matrix multiplication, etc.

[0139] Step S26: The fusion result is input into the second sleep classification model to obtain the second classification result output by the second sleep classification model whether the personnel target in the current frame is in a sleep state and the corresponding sleep classification score.

[0140] It can be understood that the model classification is performed using the fusion features, for example, 9 upper body skeletal points are obtained to obtain a 1x72 skeletal point position relationship feature vector, and then the skeletal point position relationship feature vector and the 1x256 image feature vector are spliced and combined in the form of concat to obtain a corresponding combination result, i.e. the fusion result. The result is input into the second sleep classification model composed of two fully connected layers and a Softmax layer to obtain the classification result and the corresponding sleep classification score output by the second sleep classification model whether the personnel target is in a sleep state or a non-sleep state, i.e. P Class，sleep，i .

[0141] Step S27: Based on the sleep score, the static score and the sleep classification score, a second confidence score of the personnel target in the current frame in a sleep state is determined.

[0142] It can be understood that if the static score and the sleep classification score are both greater than a third preset threshold, a first weight sum result of the static score and the sleep classification score is determined as a second confidence score of the personnel target in the current frame in a sleep state; if the static score and the sleep classification score are both greater than a fourth preset threshold and not greater than the third preset threshold, a second confidence score of the personnel target in the current frame in a sleep state is determined based on the sleep score and the first weight sum result; and if the static score and the sleep classification score are not both greater than the fourth preset threshold, the second confidence score of the personnel target in the current frame in a sleep state is set to zero. Specifically, the second confidence score of the personnel target in the current frame in a sleep state is determined based on the sleep score and the first weight sum result, which can include: a product result of the sleep score and the first weight sum result is determined as the second confidence score of the personnel target in the current frame in a sleep state. For example, the sleep score, the static score and the sleep classification score are weighted and summarized by using a weighted sum to obtain the second confidence score of the personnel target in the current frame in a sleep state, i.e. P' sleep，i The related calculation formula is as follows:

[0143]

[0144] wherein, the hyperparameter γ1 is the weight of the static score P static，i and the sleep classification score P Class，sleep，i , and the default setting of γ1 is 0.7, i.e. γ1 = 0.7.

[0145] When the static score is greater than 0.8, i.e. P static，i > 0.8, and the sleep classification score is also greater than 0.8, i.e. P class，sleep，i > 0.8, the second confidence score of the personnel target in the current frame in a sleep state is the weight sum result of the static score and the sleep classification score.

[0146] When the static score and the sleep classification score are both between 0.6 and 0.8, the judgment of the distance relationship between the key points of the target is introduced.

[0147] Moreover, only when the ratio of the distance dis AB from the head point A to the neck point B to the distance dis BC from the neck point B to the waist point C is between 1.8 and 2.0, i.e. , the second confidence score of the personnel target in the current frame in a sleep state is not zero, i.e. P' sleep，i ≠ 0.

[0148] When both the static score and the sleep state classification score are less than 0.6, the second confidence score of the personnel target in the current frame being in a sleep state is zero, that is, P' sleep，i = 0, and it can be directly determined that the personnel target in the current frame is in a non-sleep state.

[0149] Step S28: determining whether the personnel target in the current frame is in a sleep state based on the second confidence score.

[0150] In this embodiment, after the second confidence score of the personnel target in the current frame being in a sleep state is calculated, whether the personnel target in the current frame is in a sleep state can be determined based on the second confidence score.

[0151] The specific content of steps S21 to S23 can refer to the corresponding content disclosed in the foregoing embodiments, which will not be described here again.

[0152] It can be seen that, in the embodiments of the present application, the shooting angle is recognized based on the upper body region image of the human body, and then different recognition strategies are used for the target personnel in different shooting angles, so that the human body upper body skeleton key point coordinate sequence and the human body upper body image feature information can be fully utilized for the recognition of the sleep state of the personnel target, and the recognition accuracy of the sleep state of the personnel target in the back shooting of the camera can be improved.

[0153] Referring to Figure 8 The embodiments of the present application disclose a specific sleep state recognition method, which is further described and optimized compared with the previous embodiment.

[0154] It should be pointed out that, since the personnel sleep state is a long-time continuous state, the above-mentioned voting mechanism based on time sequence frames is used to analyze the results of each video frame, and the continuous multi-frame recognition results are comprehensively judged, so as to avoid the sleep state misjudgment caused by the instantaneous human body shape.

[0155] Step S31: selecting an initial video frame sequence of a preset sequence length for sleep state recognition, and determining a single-frame target state switching proportion of the initial video frame sequence; each video frame in the initial video frame sequence is a video frame with a marked sleep state recognition result.

[0156] In this embodiment, for one sleep state recognition, an initial video frame sequence of a preset sequence length for sleep state recognition can be selected, and then a single-frame target state switching proportion of the initial video frame sequence is determined.

[0157] Step S32: re-determining the sequence length of the initial video frame sequence based on a second comparison result of the single-frame target state switching proportion and a preset state switching proportion threshold to obtain a corresponding target video frame sequence.

[0158] It is understandable that frequent state transitions within a single frame of the target during the voting sequence may be caused by factors such as frequent target movement or algorithm recognition errors. Therefore, during periods of frequent state transitions, the false alarms caused by these factors can be reduced by extending the state sequence length. For example, the initial judgment duration can be set to 30 seconds, meaning that every 150 frames of video images are selected as a judgment sequence for sleep detection, and the sleep state transition frame is used as the initial frame of this judgment sequence. Then, the percentage P of single-frame target state transitions in this judgment sequence is determined. status，ratio When the proportion of target state switching in a single frame exceeds the preset state switching proportion threshold, i.e., P status，ratio >P status，th When this happens, the length of the judgment sequence is increased by (1+P). status，ratio ) times, such as Figure 9 As shown, 1 represents sleeping on duty, and 0 represents not sleeping on duty. When the target state switching ratio in a single frame is 0.4, that is, P status，ratio When the value is 0.4, the sequence length of the judgment sequence becomes 210 frames. Within the judgment sequence, the sleep status of the personnel target is... sleep After a switch occurs, and there is no single-frame state switch for 30 consecutive frames, the judgment sequence length for sleep detection is restored to the default length.

[0159] Step S33: Determine the abnormal single-frame sleeping identification result in the target video frame sequence according to the preset discrimination rule, and correct the single-frame sleeping identification result to obtain the corrected target video frame sequence.

[0160] In this embodiment, after determining the sequence length of the judgment sequence to obtain the target video frame sequence, the abnormal single-frame sleeping guard identification result in the target video frame sequence is determined according to a preset discrimination rule, and the single-frame sleeping guard identification result is corrected to obtain the corrected target video frame sequence. It is understood that since the state of a person being asleep or not asleep is a relatively stable state over a long period, the state transition of a single frame within the sequence is highly likely due to misidentification by the algorithm. Therefore, this application establishes a local feedback mechanism by incorporating the contextual information of multiple frames to evaluate and correct the state transition of a single frame.

[0161] In this embodiment, the single-frame sleeping identification result with an anomaly in the target video frame sequence is determined according to a preset discrimination rule. The specific discrimination method is as follows: Figure 10 As shown, the discrimination method is as follows:

[0162] Step S331: Determine the sleeping state switching frame of a single frame in the target video frame sequence, and select a second preset number of target video frames before and after the sleeping state switching frame as a reference; the target video frame is a video frame that is not a single frame state switching frame.

[0163] Step S332: Determine the average confidence scores of the target video frames before and after the second preset number of frames before and after the sleep status switching frame, and obtain the corresponding average first confidence score and average second confidence score.

[0164] Step S333: Determine the second weighted sum between the mean of the first confidence score and the mean of the second confidence score, and determine the second weighted sum as the target confidence score for the predicted state of the sleep state switching frame to be the sleep state.

[0165] Step S334: If the target confidence score is greater than the second confidence score corresponding to the sleep status switching frame, then the sleep status identification result corresponding to the sleep status switching frame is determined to be an abnormal single-frame sleep status identification result.

[0166] Step S335: If the target confidence score is not greater than the second confidence score corresponding to the sleep status switching frame, then the sleep status identification result corresponding to the sleep status switching frame is determined to be a non-abnormal single-frame sleep status identification result.

[0167] For example, select the state information of 5 frames before and after the single frame that switches from sleeping to guard status. When selecting these 5 frames, exclude the single frame that switches from sleeping to guard status. Then calculate the average confidence score of these 5 frames, i.e., P. status，sleep，front And calculate the mean confidence score of the last 5 frames, i.e., P status，sleep，back And the state score Ps of the sleep status switching frame. tatus，move This is the second confidence score corresponding to the sleep status switching frame, i.e., P′. sleep，i When the state score corresponding to the sleep status switching frame is less than the weighted sum of the confidence scores of the preceding and following 5 frames, i.e., P status，move <P status，sleep When the recognition result of the intermediate frame is deemed abnormal, it is corrected to ensure consistency with the preceding and following frames. The second confidence score of this intermediate frame can then be set to 0.8, where P... status，sleep This represents the weighted sum of the average confidence scores from the preceding and following 5 frames. For example... Figure 11 As shown, the recognition result for the first 5 frames is "sleeping on duty," while the recognition results for the 6th and 8th frames are "not sleeping on duty." That is, the 6th and 8th frames are both "sleeping on duty" state switching frames. The recognition results for the 7th, 9th to 14th frames are also "sleeping on duty." Therefore, the formula for calculating the average confidence score of the 5 frames before and after the 6th frame is as follows:

[0168]

[0169]

[0170] Correspondingly, the weight sum calculation formula of the average of the confidence scores of the five frames before and after the sixth frame is as follows:

[0171] P status，sleep = 0.4 * P status，sleep，front + 0.6 * P status，sleep，back = 0.838.

[0172] The state score of the sixth frame is the second confidence score corresponding to the sixth frame, that is:

[0173] P status，move = P' sleep，6 = 0.6.

[0174] It can be seen that P status，move < P status，sleep , it is determined that the recognition result of the sixth frame is abnormal, and the recognition result of the sixth frame is corrected, as shown in the following formula: Figure 12 The recognition result of the sixth frame is adjusted to sleep state, and the confidence score of the sixth frame is modified to 0.8.

[0175] It can be understood that the dynamic planning of the video sequence length and the correction of the single frame recognition result are introduced, so that the false detection rate can be greatly reduced.

[0176] Step S34: Determine the total number of frames in the modified target video frame sequence, and determine the number of frames in the modified target video frame sequence whose sleep recognition result is sleep state.

[0177] Step S35: If the ratio of the frame number to the total frame number is greater than the preset frame proportion threshold, and the sleep recognition result corresponding to the first preset number of continuous video frames in the modified target video frame sequence is all sleep state, it is determined that the personnel target in the initial video frame sequence is in sleep state, and a corresponding alarm operation is triggered.

[0178] Step S36: If the ratio of the frame number to the total frame number is not greater than the preset frame proportion threshold and / or the sleep recognition result corresponding to the first preset number of continuous video frames in the modified target video frame sequence is not all sleep state, it is determined that the personnel target in the initial video frame sequence is in non-sleep state.

[0179] For example, when the ratio of the number of frames marked as sleep state in the modified target video sequence to the total number of frames in the sequence is greater than the preset frame proportion threshold, and the sleep state of the last 20% of the frames in the modified target video sequence is continuous, it is determined that the personnel target is in sleep state, and a violation behavior alarm is triggered, that is:

[0180]

[0181] wherein the preset frame proportion threshold is set as 0.8, i.e. P sleep，ration，th = 0.8, assuming that the sequence length of the target video sequence after correction is 150 frames, when the ratio of the frame number of the target video sequence in which the flag is in the sleep state to the total frame number is greater than the preset frame proportion threshold, i.e. P sleep，ratio，sum > P sleep，ratio，th = 0.8, and the recognition result of the last 30 frames of the sequence is all in the sleep state, it is determined that the personnel target in the sequence is in the sleep state, and a violation behavior alarm is triggered.

[0182] It can be understood that for the difficult-to-distinguish camera backshot target, the technical solution of the present application combines the recognition results of the fusion feature classification and the spatial information logical judgment between the skeletal key points, and makes a comprehensive judgment on whether the camera backshot target is in the sleep state based on the time sequence information of multiple frames, which greatly improves the recognition accuracy of the backshot sleep behavior. Among them, by combining the confidence of each skeletal key point of the target detection model, the positional relationship, i.e. distance relationship and angle relationship, between each key point of the same personnel target, and the motion law of each skeletal key point in the time dimension, the abnormal value of the skeletal key point positioning result can be effectively filtered to ensure the accuracy of the skeletal point positioning result. Based on the time sequence motion law of the personnel skeletal key points, the key point prediction method is used to correct the abnormal key points, i.e. to replace the abnormal coordinates with the predicted coordinates, so as to reduce the influence of the abnormal key points on the recognition result. In the technical solution of the present application, the sleep behavior time sequence judgment logic is optimized. Not only is the local feedback mechanism that relates to the context recognition result, i.e. the recognition result of the context multiple frames of sleep behavior, introduced to correct and judge the abnormal single frame prediction result, avoiding the abnormal error of single frame recognition, but also the adaptive adjustment of the time sequence length is adopted, which can flexibly adjust the time sequence judgment time, can ensure the recognition accuracy of the algorithm, shorten the recognition time, and speed up the alarm speed of the sleep violation behavior.

[0183] It can be seen that in the embodiment of the present application, the shooting angle is recognized based on the image of the upper body region of the human body, and then different recognition strategies are used for the target personnel in different shooting angles to determine the single frame personnel target sleep state recognition, which can improve the recognition accuracy of the personnel target sleep state in the single frame under the condition of camera backshot. Then, based on the time sequence characteristics of the sleep behavior, a local feedback mechanism is used to relate the recognition results of multiple frames to finally determine whether the behavior of the personnel target is a violation sleep behavior. The use of the time sequence feedback mechanism to correct the abnormal frames in the video frame sequence can further ensure the recognition accuracy of the sleep violation behavior.

[0184] Correspondingly, the present application also discloses a sleep recognition device, as shown in Figure 13 The device comprises:

[0185] The first detection module 11 is configured to input an original image of a current frame collected by a camera into a pre-constructed target detection model to obtain a human upper body bounding box and a human upper body skeleton key point coordinate sequence in the original image output by the target detection model;

[0186] The cropping module 12 is configured to crop the original image according to the human upper body bounding box to obtain a human upper body region image;

[0187] The second detection module 13 is configured to input the human upper body region image into a residual network to obtain a shooting angle of the camera relative to a target person identified by the residual network;

[0188] The first sleep state recognition module 14 is configured to, when the shooting angle is front shooting, side shooting or top shooting, input the target person into a first sleep state classification model to obtain a first classification result of whether the target person in the current frame is in a sleep state output by the first sleep state classification model;

[0189] The second sleep state recognition module 15 is configured to, when the shooting angle is back shooting, calculate a skeleton point spatial relationship by using the human upper body skeleton key point coordinate sequence, and comprehensively judge whether the target person in the current frame is in a sleep state by using the skeleton point spatial relationship and the image feature output by the residual network.

[0190] As can be seen from the above, in the embodiments of the present application, the shooting angle is identified based on the human upper body region image, and then different recognition strategies are adopted for the target person in different shooting angles, so that the human upper body skeleton key point coordinate sequence and the human upper body image feature information can be fully utilized to identify the sleep state of the target person, and the recognition accuracy of the sleep state of the target person in the back shooting of the camera can be improved.

[0191] In some specific embodiments, the sleep state recognition device can further include:

[0192] The confidence determination module is configured to determine the confidence of each skeleton key point based on a first confidence score corresponding to each skeleton key point output by the target detection model;

[0193] The first determination module is configured to determine that the skeleton key point is an abnormal point if the confidence is less than a pre-set confidence filtering threshold, and replace the abnormal coordinates of the abnormal point with predicted coordinates of the abnormal point determined based on a skeleton key point prediction rule;

[0194] The second determination module is configured to determine that the skeleton key point is a non-abnormal point if the confidence is not less than the pre-set confidence filtering threshold.

[0195] In some specific embodiments, the confidence determination module can specifically include:

[0196] The first determination unit is configured to determine the first confidence score as the confidence of the skeletal key point if the first confidence score corresponding to the skeletal key point output by the target detection model is not less than a first preset threshold value.

[0197] The second determination unit is configured to determine a probability that the skeletal key point is the non-anomalous point and calculate the confidence of the skeletal key point according to the first confidence score and the probability if the first confidence score is greater than a second preset threshold value and less than the first preset threshold value.

[0198] The third determination unit is configured to determine a position relationship score of the skeletal key point and determine a probability that the skeletal key point is the non-anomalous point, and calculate the confidence of the skeletal key point according to the first confidence score, the position relationship score and the probability if the first confidence score is not greater than the second preset threshold value.

[0199] In some specific embodiments, the second determination unit can specifically include:

[0200] The position prediction range determination sub-unit is configured to determine a distance prediction range and an angle prediction range of the skeletal key point according to a preset key point prediction method based on the prediction confidence of the skeletal key point.

[0201] The probability determination sub-unit is configured to calculate the probability that the skeletal key point is the non-anomalous point according to the distance prediction range and the angle prediction range.

[0202] In some specific embodiments, the third determination unit can specifically include:

[0203] The position relationship determination sub-unit is configured to determine a position relationship between the skeletal key points of the personnel target; the position relationship includes a distance relationship and an angle relationship.

[0204] The anomalous point judgment sub-unit is configured to judge whether the skeletal key point is the anomalous point based on the position relationship to obtain a corresponding first judgment result.

[0205] The position relationship score determination sub-unit is configured to determine a position relationship score of the skeletal key point based on the first judgment result; the position relationship score includes a distance relationship score and an angle relationship score.

[0206] In some specific embodiments, the second sleep state recognition module 15 can specifically include:

[0207] The sleep state score and the still state score determination submodule is configured to determine a sleep state score of the personnel target in the current frame based on the spatial relationship of the skeleton points and a still state score of the personnel target in the current frame.

[0208] The fusion submodule is configured to fuse the spatial relationship of the skeleton points and the image features output by the residual network to obtain a corresponding fusion result.

[0209] The sleep state classification score determination submodule is configured to input the fusion result into a second sleep state classification model to obtain a second classification result of whether the personnel target in the current frame is in a sleep state and a corresponding sleep state classification score output by the second sleep state classification model.

[0210] The second confidence score determination submodule is configured to determine a second confidence score of whether the personnel target in the current frame is in a sleep state based on the sleep state score, the still state score and the sleep state classification score.

[0211] The second sleep state recognition submodule is configured to determine whether the personnel target in the current frame is in a sleep state based on the second confidence score.

[0212] In some specific embodiments, the second sleep state recognition module 15 can specifically include:

[0213] The distance determination submodule is configured to calculate a first distance between a head key point and a neck key point and a second distance between the neck key point and a waist key point by using the upper body skeleton key point coordinate sequence, and determine a displacement distance of each skeleton key point between adjacent frames.

[0214] In some specific embodiments, the sleep state score and the still state score determination submodule can specifically include:

[0215] The ratio determination unit is configured to determine a ratio between the first distance and the second distance.

[0216] The condition determination unit is configured to determine whether the ratio satisfies a preset condition to obtain a corresponding second determination result.

[0217] The sleep state score determination unit is configured to determine a sleep state score of the personnel target in the current frame based on the second determination result.

[0218] The still state score determination unit is configured to determine a still state score of each skeleton key point in a still state based on a first comparison result of the displacement distance of each skeleton key point between adjacent frames and a preset displacement threshold, and calculate a still score of the personnel target in the current frame in the still state based on the still state score of each skeleton key point.

[0219] In some specific embodiments, the second confidence score determination sub-module can specifically include:

[0220] The fourth determination unit is configured to determine, if the static score and the sleep state classification score are both greater than a third preset threshold, a first weight sum result of the static score and the sleep state classification score as a second confidence score of the personnel target in the current frame being in a sleep state;

[0221] The fifth determination unit is configured to determine, if the static score and the sleep state classification score are both greater than a fourth preset threshold and not greater than the third preset threshold, a second confidence score of the personnel target in the current frame being in a sleep state based on the sleep state score and the first weight sum result;

[0222] The sixth determination unit is configured to set the second confidence score of the personnel target in the current frame being in a sleep state to zero, if the static score and the sleep state classification score are both not greater than the fourth preset threshold.

[0223] In some specific embodiments, the sleep state recognition device can further include:

[0224] The video frame sequence selection module is configured to select an initial video frame sequence for sleep state recognition with a preset sequence length;

[0225] The state switching proportion determination module is configured to determine a single-frame target state switching proportion of the initial video frame sequence; each video frame in the initial video frame sequence is a video frame with a marked sleep state recognition result;

[0226] The target video frame sequence determination module is configured to determine a corresponding target video frame sequence based on a second comparison result of the single-frame target state switching proportion and a preset state switching proportion threshold, by re-determining a sequence length of the initial video frame sequence;

[0227] The abnormal recognition result determination module is configured to determine, according to a preset discrimination rule, a single-frame sleep state recognition result in the target video frame sequence that is abnormal;

[0228] The abnormal recognition result correction module is configured to correct the single-frame sleep state recognition result to obtain a corrected target video frame sequence.

[0229] The frame number determination module is configured to determine a total frame number in the corrected target video frame sequence, and determine a frame number in the corrected target video frame sequence in which the sleep state recognition result is a sleep state;

[0230] The third determination module is configured to determine that the personnel target in the initial video frame sequence is in a sleep state and trigger an alarm operation if the ratio of the frame number to the total frame number is greater than the preset frame proportion threshold and all the sleep state recognition results corresponding to the first preset number of continuous video frames in the modified target video frame sequence are sleep state.

[0231] The fourth determination module is configured to determine that the personnel target in the initial video frame sequence is in a non-sleep state if the ratio of the frame number to the total frame number is not greater than the preset frame proportion threshold and / or the sleep state recognition results corresponding to the first preset number of continuous video frames in the modified target video frame sequence are not all sleep state.

[0232] Further, the embodiment of the present application further provides an electronic device. Figure 14 The electronic device 20 shown in the figure is not considered as any limitation on the use range of the present application according to an exemplary embodiment.

[0233] Figure 14 The electronic device 20 provided by the embodiment of the present application is shown in the structural schematic diagram. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is used to store a computer program, the computer program is loaded and executed by the processor 21 to realize the related steps in the sleep state recognition method disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the embodiment can be an electronic computer.

[0234] In the embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited here; the input / output interface 25 is used to obtain external input data or output data to the outside world, and the specific interface type can be selected according to the specific application needs, which is not limited here.

[0235] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0236] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. The computer program 222 can further include computer programs capable of performing other specific work in addition to the computer program capable of performing the sleep state recognition method executed by the electronic device 20 disclosed in any of the foregoing embodiments.

[0237] Further, the embodiments of the present application further disclose a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the steps of the sleep state recognition method disclosed in any of the foregoing embodiments.

[0238] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

Claims

1. A method for identifying sleep deprivation on duty, characterized in that, include: The original image of the current frame captured by the camera is input into the pre-built target detection model to obtain the human upper body detection box and the human upper body skeleton key point coordinate sequence in the original image output by the target detection model. The original image is cropped according to the human upper body detection box to obtain the human upper body region image, and the human upper body region image is input into the residual network to obtain the shooting angle of the camera relative to the human target as recognized by the residual network. If the shooting angle is a front shot, a side shot, or a top shot, the personnel target is input into the first sleeping post classification model to obtain the first classification result of whether the personnel target in the current frame is in a sleeping post state, which is output by the first sleeping post classification model; If the shooting angle is a back shot, the spatial relationship of the skeletal points is calculated using the coordinate sequence of the key points of the upper body skeleton, and the spatial relationship of the skeletal points and the image features output by the residual network are used to comprehensively determine whether the target person in the current frame is in a sleeping position. The step of comprehensively determining whether the personnel target in the current frame is in a sleeping state by utilizing the spatial relationship of the skeletal points and the image features output by the residual network includes: Based on the spatial relationship of the skeleton points, the sleeping score of the personnel target in the current frame when it is in a sleeping state and the stationary score of the personnel target in the current frame when it is in a stationary state are determined. The spatial relationship of the skeletal points and the image features output by the residual network are fused to obtain the corresponding fusion result; The fusion result is input into the second sleep deprivation classification model to obtain the second classification result of whether the personnel target in the current frame is in a sleep deprivation state and its corresponding sleep deprivation classification score. Based on the sleeping post score, the stillness score, and the sleeping post classification score, a second confidence score is determined to indicate that the personnel target in the current frame is in a sleeping post state; Based on the second confidence score, it is determined whether the personnel target in the current frame is in a sleeping state.

2. The method for identifying sleeping on duty according to claim 1, characterized in that, Before calculating the spatial relationship of skeletal points using the coordinate sequence of key points of the upper body skeleton, the method further includes: The confidence level of each skeletal key point is determined based on the first confidence score corresponding to each skeletal key point output by the target detection model. If the confidence level is less than the preset confidence filtering threshold, the skeletal key point is determined to be an anomaly, and the anomaly coordinates of the anomaly are replaced by the predicted coordinates of the anomaly determined based on the skeletal key point prediction rules. If the confidence level is not less than the preset confidence filtering threshold, then the skeletal key point is determined to be a non-abnormal point.

3. The method for identifying sleeping on duty according to claim 2, characterized in that, The determination of the confidence level of each skeletal keypoint based on the first confidence score corresponding to each skeletal keypoint output by the target detection model includes: If the first confidence score corresponding to the skeletal key point output by the target detection model is not less than the first preset threshold, then the first confidence score is determined as the confidence score of the skeletal key point. If the first confidence score is greater than the second preset threshold and less than the first preset threshold, then the probability that the skeletal key point is the non-abnormal point is determined, and the confidence of the skeletal key point is calculated based on the first confidence score and the probability. If the first confidence score is not greater than the second preset threshold, then the positional relationship score of the skeletal key point and the probability of determining the skeletal key point as the non-abnormal point are determined, and the confidence of the skeletal key point is calculated based on the first confidence score, the positional relationship score and the probability.

4. The method for identifying sleep deprivation during work hours according to claim 3, characterized in that, The probability of determining the skeletal key point as a non-abnormal point includes: Based on the prediction reliability of the skeletal key points, the distance prediction range and angle prediction range of the skeletal key points are determined according to the preset key point prediction method. The probability that the skeletal key point is a non-abnormal point is calculated based on the distance prediction range and the angle prediction range.

5. The method for identifying sleep deprivation during work hours according to claim 3, characterized in that, The determination of the positional relationship score of the skeletal key points includes: Determine the positional relationships between the skeletal key points of the personnel target; the positional relationships include distance relationships and angular relationships; Based on the positional relationship, it is determined whether the skeletal key point is an abnormal point to obtain the corresponding first judgment result; Based on the first judgment result, the positional relationship score of the skeletal key points is determined; the positional relationship score includes distance relationship score and angle relationship score.

6. The method for identifying sleep deprivation during work hours according to claim 1, characterized in that, The calculation of the spatial relationship of skeletal points using the coordinate sequence of key points of the upper body skeleton includes: The first distance between the head key point and the neck key point and the second distance between the neck key point and the waist key point are calculated using the upper body skeleton key point coordinate sequence, and the displacement distance of each skeleton key point between two adjacent frames is determined. Accordingly, determining the sleeping score (when the personnel target in the current frame is in a sleeping state) and the stationary score (when the personnel target in the current frame is in a stationary state) based on the spatial relationship of the skeletal points includes: Determine the ratio between the first distance and the second distance; Determine whether the ratio meets the preset conditions to obtain the corresponding second judgment result, and determine the sleep score of the personnel target in the current frame in the sleep state based on the second judgment result; Based on the first comparison result between the displacement distance of each skeletal key point between two adjacent frames and a preset displacement threshold, a static state score is determined for each skeletal key point in a static state, and a static state score for the person target in the current frame in a static state is calculated based on the static state score corresponding to each skeletal key point.

7. The method for identifying sleep deprivation during work hours according to claim 1, characterized in that, The second confidence score for determining whether the personnel target in the current frame is in a sleeping position based on the sleeping position score, the stillness score, and the sleeping position classification score includes: If both the static score and the sleeping-on-duty classification score are greater than the third preset threshold, then the first weighted sum of the static score and the sleeping-on-duty classification score is determined as the second confidence score that the personnel target in the current frame is in a sleeping-on-duty state. If both the static score and the sleeping-on-duty classification score are greater than the fourth preset threshold and not greater than the third preset threshold, then a second confidence score is determined based on the sleeping-on-duty score and the sum of the first weights to determine whether the person target in the current frame is in a sleeping-on-duty state. If neither the static score nor the sleeping-on-duty classification score is greater than the fourth preset threshold, then the second confidence score of the personnel target in the current frame being in a sleeping-on-duty state is set to zero.

8. The method for identifying sleep deprivation during work hours according to claim 7, characterized in that, Also includes: Select an initial video frame sequence of a preset sequence length for sleep detection, and determine the percentage of target state switching per frame in the initial video frame sequence. Each video frame in the initial video frame sequence is a video frame with the sleeping post identification result already marked; Based on the second comparison result between the single-frame target state switching ratio and the preset state switching ratio threshold, the sequence length of the initial video frame sequence is re-determined to obtain the corresponding target video frame sequence; According to the preset discrimination rules, the abnormal single-frame sleeping guard identification results in the target video frame sequence are determined, and the single-frame sleeping guard identification results are corrected to obtain the corrected target video frame sequence. Determine the total number of frames in the corrected target video frame sequence, and determine the number of frames in the corrected target video frame sequence where the sleeping post identification result is a sleeping post state; If the ratio of the number of frames to the total number of frames is greater than the preset frame percentage threshold and the corrected target video frame sequence contains consecutive video frames of a first preset number of frames whose corresponding sleep-on-duty identification results are all in the sleep-on-duty state, then it is determined that the personnel target in the initial video frame sequence is in the sleep-on-duty state, and the corresponding alarm operation is triggered. If the ratio of the number of frames to the total number of frames is not greater than the preset frame percentage threshold and / or the sleeping identification results corresponding to the first preset number of consecutive video frames in the corrected target video frame sequence are not all in a sleeping state, then the personnel target in the initial video frame sequence is determined to be in a non-sleeping state.

9. A device for detecting sleep deprivation on duty, characterized in that, include: The first detection module is used to input the original image of the current frame captured by the camera into the pre-built target detection model to obtain the human upper body detection box and the human upper body skeleton key point coordinate sequence in the original image output by the target detection model. The cropping module is used to crop the original image according to the human upper body detection box to obtain an image of the human upper body region; The second detection module is used to input an image of the upper body region of the human body into a residual network to obtain the shooting angle of the camera relative to the human target as identified and output by the residual network. The first sleeping post recognition module is used to input the personnel target into the first sleeping post classification model when the shooting angle is front shot, side shot or top shot, and obtain the first classification result of whether the personnel target in the current frame of the first sleeping post classification model is in a sleeping post state. The second sleeping-on-duty recognition module is used to calculate the spatial relationship of the skeletal points using the coordinate sequence of the key points of the upper body skeleton when the shooting angle is a back shot, and to comprehensively determine whether the target person in the current frame is in a sleeping-on-duty state using the spatial relationship of the skeletal points and the image features output by the residual network. The second sleep deprivation detection module includes: The Sleeping Score and Stillness Score Determination Submodule is used to determine the Sleeping Score when the personnel target in the current frame is in a sleeping state and the Stillness Score when the personnel target in the current frame is in a still state, based on the spatial relationship of the skeleton points. The fusion submodule is used to fuse the spatial relationship of the skeleton points and the image features output by the residual network to obtain the corresponding fusion result; The sleeping-on-duty classification score determination submodule is used to input the fusion result into the second sleeping-on-duty classification model to obtain the second classification result of whether the personnel target in the current frame is in a sleeping-on-duty state and its corresponding sleeping-on-duty classification score, which is output by the second sleeping-on-duty classification model. The second confidence score determination submodule is used to determine a second confidence score that the personnel target in the current frame is in a sleeping state based on the sleeping score, the stillness score and the sleeping classification score; The second sleep-on-duty identification submodule is used to determine whether the personnel target in the current frame is in a sleep-on-duty state based on the second confidence score.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the sleep deprivation identification method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the sleep deprivation identification method as described in any one of claims 1 to 8.

Images