Sleep post detection method and device, electronic equipment and storage medium

Abstract

The invention provides a sleep post detection method and device, electronic equipment and a storage medium, and relates to the technical field of video surveillance, and the method comprises the steps: obtaining at least two frames of to-be-detected video images in a surveillance video stream, and extracting a head and shoulder region graph of a to-be-detected person in each frame of to-be-detected video image; determining the sleep confidence coefficient of each head and shoulder area graph by using a sleep classification model; wherein the sleep post classification model is obtained by performing comparison training between an initial sleep post classification model and a pre-trained image feature extraction model on the initial sleep post classification model based on a sample image pair and the pre-trained image feature extraction model; the initial sleep post classification model is obtained by replacing a first convolutional layer of a pre-trained image feature extraction model with a deformable convolutional layer; and determining whether the to-be-detected person is in a sleep state based on the sleep confidence coefficient. According to the technical scheme provided by the invention, false detection and missing detection of sleep detection can be reduced, and the accuracy and recall rate of sleep detection are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video surveillance, and in particular to a method, device, electronic equipment and storage medium for detecting sleeping on duty. Background Art

[0002] Sleeping on duty refers to the on-duty personnel sleeping on duty for various reasons, mainly by lying down, leaning or sleeping. Sleeping on duty can easily cause accidents and bring safety hazards to society and enterprises. Therefore, it is very important to promptly detect and correct the on-duty personnel's sleeping behavior.

[0003] With the development of machine vision, in related technologies, a trained sleeping post classification model can be used to determine sleeping post behavior. However, the sleeping post classification model needs to resample the sample images to the same size during training. The human body posture information will be lost or changed during the image stretching process, which reduces the distinction between sleeping posts and non-sleeping posts, resulting in false detection and missed detection in sleeping post detection. Summary of the invention

[0004] The invention provides a sleep-post detection method, device, electronic equipment and storage medium, which are used to solve the problems of false detection and missed detection in the sleep-post detection in the prior art and improve the accuracy and recall rate of the sleep-post detection.

[0005] The present invention provides a method for detecting a sleeping post, comprising:

[0006] Obtain at least two frames of video images to be detected in the monitoring video stream, and extract the head and shoulder area map of the person to be detected in each frame of the video image to be detected;

[0007] Determine the confidence of each head-shoulder region image using a head-shoulder classification model; wherein the head-shoulder classification model is obtained by performing comparative training between the initial head-shoulder classification model and the pre-trained image feature extraction model based on the sample image pair and the pre-trained image feature extraction model, and the initial head-shoulder classification model is obtained by replacing the first convolution layer of the pre-trained image feature extraction model with a deformable convolution layer;

[0008] Based on the sleeping-on-duty confidence level, it is determined whether the person to be detected is in a sleeping-on-duty state.

[0009] According to a method for detecting a sleeping post provided by the present invention, the sleeping post classification model is obtained by training based on the following steps:

[0010] Acquire the sample image pair, the sample image pair comprising a first sample image and a second sample image, the second sample image being obtained by resampling all images in the first sample image to the same size; the first sample image and the second sample image correspond one to one;

[0011] Extracting a first image feature vector of the first sample image based on the pre-trained image feature extraction model;

[0012] Based on the initial image classification model, extracting a second image feature vector of the second sample image corresponding to the first sample image, and determining a sample image confidence level of the second sample image;

[0013] Determine a loss function value based on the first image feature vector, the second image feature vector and the sample confidence level;

[0014] The model parameters of the initial sleep-station classification model are adjusted based on the loss function value until the loss function value is minimized to obtain the sleep-station classification model.

[0015] According to a method for detecting a sleeping position provided by the present invention, the loss function value is determined based on the first image feature vector, the second image feature vector and the sample sleeping position confidence, including:

[0016] determining a relative entropy of the first image feature vector and the second image feature vector;

[0017] Determine the cross entropy of the second sample image based on the sample confidence level and the sample label value of the second sample image;

[0018] The loss function value is determined based on the relative entropy, the first weight of the relative entropy, the cross entropy and the second weight of the cross entropy; wherein the sum of the weight value of the first weight and the weight value of the second weight is 1.

[0019] According to a post sleeping detection method provided by the present invention, the weight value of the first weight decreases as the number of training rounds increases.

[0020] A method for detecting a sleeping post provided by the present invention further comprises:

[0021] Based on the training rounds, a weight value of the first weight is determined using a cosine annealing algorithm.

[0022] According to a method for detecting a person sleeping on duty provided by the present invention, determining whether the person to be detected is in a state of sleeping on duty based on the confidence level of the person sleeping on duty comprises:

[0023] Determining the mean and standard deviation of the confidence level of the sleep post;

[0024] When the average value is greater than the first threshold value and the standard deviation is less than the second threshold value, it is determined that the person to be detected is in a sleeping state.

[0025] According to a sleep guard detection method provided by the present invention, the sleep guard classification model is used to determine the sleep guard confidence of each head and shoulder area graph, including:

[0026] Determine the intersection-and-union ratio of the head-shoulder region images of two adjacent frames;

[0027] When the intersection-and-union ratios corresponding to the at least two frames of video images to be detected are all greater than a third threshold, it is determined that the person to be detected is in a stationary state;

[0028] When it is determined that the person to be detected is in a stationary state, the sleeping position classification model is used to determine the sleeping position confidence level of each head-shoulder region image.

[0029] The present invention also provides a sleeping post detection device, comprising:

[0030] An image acquisition module is used to acquire at least two frames of video images to be detected in the monitoring video stream, and extract a head and shoulder area image of the person to be detected in each frame of the video image to be detected;

[0031] An image classification module, used for determining the confidence of each head-shoulder region image by using a head-shoulder classification model; wherein the head-shoulder classification model is obtained by performing comparative training between the initial head-shoulder classification model and the pre-trained image feature extraction model on an initial head-shoulder classification model based on a sample image pair and a pre-trained image feature extraction model, and the initial head-shoulder classification model is obtained by replacing the first convolution layer of the pre-trained image feature extraction model with a deformable convolution layer;

[0032] The sleeping-on-duty determination module is used to determine whether the person to be detected is in a sleeping-on-duty state based on the sleeping-on-duty confidence level.

[0033] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for detecting a sleeping post as described above is implemented.

[0034] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned methods for detecting sleeping on the post is implemented.

[0035] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for detecting sleeping on the post.

[0036] The method, device, electronic device and storage medium for detecting sleeping on the post provided by the present invention extract the head-shoulder area map of the person to be detected from multiple frames of video images to be detected in a monitoring video stream, and use the head-shoulder area map to first determine whether the person to be detected is in a stationary state. If the person is in a stationary state, a sleeping post classification model is used to determine the sleeping post confidence of each head-shoulder area map, and then based on the sleeping post confidence, it is determined whether the person to be detected is in a sleeping post state, thereby realizing the sleeping post detection of the person to be detected. Among them, the sleeping post classification model is obtained by performing comparative training between the initial sleeping post classification model and the pre-trained image feature extraction model based on the sample image pairs and the pre-trained image feature extraction model. The initial sleeping post classification model is obtained by replacing the first convolution layer of the pre-trained image feature extraction model with a deformable convolution layer. In this way, through the comparative training between the initial sleeping post classification model and the pre-trained image feature extraction model and combining with the deformable convolution, the alignment of the sample image pairs can be achieved, so that the sleeping post classification model can adapt to the stretching and deformation of the image, thereby improving the ability of the sleeping post classification model to distinguish the posture details of the head and shoulder area of ​​the person to be detected, thereby improving the distinction between sleeping posts and non-sleeping posts, thereby reducing the false detection and missed detection of sleeping post detection, and improving the accuracy and recall rate of sleeping post detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 It is a schematic flow chart of a sleeping post detection method provided by an embodiment of the present invention;

[0039] Figure 2 It is a flow chart of a training method for a sleeping post classification model provided by an embodiment of the present invention;

[0040] Figure 3 It is a schematic diagram of the adjustment of the loss function by the first weight change during the training process of the sleep post classification model in an embodiment of the present invention;

[0041] Figure 4 Schematic diagram of the training principle of the sleeper classification model in an embodiment of the present invention;

[0042] Figure 5 It is one of the schematic diagrams of the sleeping post detection result in the embodiment of the present invention;

[0043] Figure 6 2 is a schematic diagram of the detection result of sleeping on the post in an embodiment of the present invention;

[0044] Figure 7 It is a structural schematic diagram of a sleeping post detection device provided by an embodiment of the present invention;

[0045] Figure 8 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] It should be noted that the serial numbers assigned to the objects described in the present invention, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning.

[0048] Combine the following Figure 1-Figure 6 The post sleeping detection method of the present invention is described. The post sleeping detection method can be applied to monitoring equipment or electronic equipment that is connected to the monitoring equipment for communication, and the electronic equipment may include a terminal device or a server, etc. Among them, the monitoring equipment may include an Internet protocol camera (IPC), a PTZ camera, etc.; the terminal device may include a mobile phone, a computer, a vehicle-mounted device, a tablet computer, a wearable device, etc.; the server may include an independent server, a cluster server or a cloud server, etc. The post sleeping detection method can also be applied to a post sleeping detection device set in an electronic device, and the post sleeping detection device can be implemented by software, hardware, or a combination of the two.

[0049] Figure 1 The schematic diagram of the process of the sleep guard detection method provided by the embodiment of the present invention is exemplified. Figure 1 As shown, the sleeping post detection method may include the following steps 110 to 130.

[0050] Step 110: Obtain at least two frames of video images to be detected in the surveillance video stream, and extract the head and shoulder area image of the person to be detected in each frame of the video image to be detected.

[0051] The camera of the monitoring device can collect the on-duty images of the person to be detected at the duty post in real time to detect the duty status of the person to be detected. Correspondingly, the electronic device can obtain the monitoring video stream of the person to be detected from the monitoring device. The electronic device can obtain video frames from the monitoring video stream, for example, it can obtain continuous N (N is an integer greater than 1) frames of video images to be detected, or extract N frames of video images to be detected at intervals, or it can obtain video images to be detected within a preset time length, and the preset time length may include at least two frames of video images to be detected, where N can be customized. For each frame of video image to be detected obtained, the electronic device can extract the head and shoulder area image of the person to be detected from the video image to be detected through target detection.

[0052] Exemplarily, the video image to be detected may include one or more persons to be detected, and the electronic device may extract the head and shoulder area map of each person to be detected in the video image to be detected, or extract the head and shoulder area map of the person to be detected specified in the video image to be detected. Exemplarily, the head and shoulder area map of each person to be detected may be marked by a person identification, each person identification corresponds to a person to be detected, and the person identification may be obtained by the electronic device numbering the positions of each person to be detected when acquiring the video image to be detected.

[0053] Step 120: Determine the confidence level of each head and shoulder area graph using the head and shoulder classification model.

[0054] Among them, the sleeping post classification model is obtained by performing comparative training between the initial sleeping post classification model and the pre-trained image feature extraction model based on the sample image pairs and the pre-trained image feature extraction model. The initial sleeping post classification model is obtained by replacing the first convolution layer of the pre-trained image feature extraction model with a deformable convolution layer.

[0055] Exemplarily, the pre-trained image feature extraction model can be a pre-trained model loaded with ImageNet pre-trained weights, and the pre-trained model can include, for example, ResNet, LeNet, VGG, AlexNet, DenseNet and other convolutional neural networks (CNN), etc., which are not limited in the embodiments of the present invention. Among them, ImageNet is a natural image training set with millions of annotations. The model trained based on the ImageNet data set has the generalization ability generally possessed by deep learning models, and can be used as a starting point to continue training the required model.

[0056] Exemplarily, the sample image pair may include a first sample image and a second sample image obtained by resampling all images in the first sample image to the same size, that is, the first sample image and the second sample image correspond one-to-one to constitute the sample image pair required for training.

[0057] The first sample image may include a first sample sleeping image and a first non-sample sleeping image, or may include a first sample sleeping head and shoulder region image and a first non-sample sleeping head and shoulder region image. Similarly, the second sample image may include a corresponding second sample sleeping image and a second non-sample sleeping image, or may include a corresponding second sample sleeping head and shoulder region image and a second non-sample sleeping head and shoulder region image.

[0058] In the process of training the initial sleep classification model, the first sample image can be input into the pre-trained image feature extraction model, and the second sample image can be input into the initial sleep classification model. The loss function value is calculated according to the output results of the two models, and the model parameters of the initial sleep classification model are adjusted based on the loss function value until the loss function value is minimized to obtain the sleep classification model. For example, the image feature vectors output by the two models can be compared, and the loss function can be calculated based on the comparison result.

[0059] For each person to be detected, the electronic device can input the head and shoulder area images of the person to be detected in multiple frames of video images to be detected into the sleeping position classification model respectively, and obtain the sleeping position confidence of each head and shoulder area image output by the sleeping position classification model.

[0060] Alternatively, the electronic device can first determine whether the person to be detected is in a stationary state based on the head and shoulder area map of the person to be detected in multiple frames of video images to be detected. If in a stationary state, the sleeping confidence of each head and shoulder area map is determined using the sleeping classification model. For example, the electronic device can determine the similarity between the head and shoulder area maps of each frame. If the similarities are greater than a preset similarity threshold, it is determined that the person to be detected is in a stationary state. Alternatively, the intersection and union ratio of the head and shoulder area maps between each adjacent frame can be determined. If the intersection and union ratios are greater than a preset intersection and union ratio threshold, it is determined that the person to be detected is in a stationary state.

[0061] Specifically, in an example embodiment, using the sleeping position classification model to determine the sleeping position confidence of each head-shoulder area image may include: determining the intersection-and-union ratio of two adjacent frames of the head-shoulder area images; when the intersection-and-union ratios corresponding to at least two frames of the video images to be detected are greater than a third threshold, determining that the person to be detected is in a stationary state; when it is determined that the person to be detected is in a stationary state, using the sleeping position classification model to determine the sleeping position confidence of each head-shoulder area image.

[0062] For example, the electronic device acquires 5 frames of video images to be detected within a preset time period. For each frame of the video image to be detected, the intersection and union ratio of the head and shoulder area map in the video image to be detected and the head and shoulder area map in the previous frame of the video image to be detected is determined. If the intersection and union ratios corresponding to the 5 frames of video images to be detected are all greater than the intersection and union ratio threshold, it can be determined that the person to be detected is in a stationary state. If the intersection and union ratio of one frame among the 5 frames of video images to be detected is less than or equal to the intersection and union ratio threshold, the video image to be detected is acquired from the beginning of the new timing from the frame of the video image to be detected and the intersection and union ratio calculation is performed again. For example, if the intersection and union ratio between the 3rd frame of the image to be detected and the 2nd frame of the image to be detected is less than the intersection and union ratio threshold, it is considered that the person to be detected is not in a stationary state at this time, and 5 frames of video images to be detected can be acquired again from the 3rd frame of the image to be detected to judge the stationary state.

[0063] If the person to be detected is in a stationary state, the sleeping post classification model is used to determine the sleeping post confidence of each head and shoulder area graph, and the sleeping post confidence can represent the probability that the person to be detected is in a sleeping post state. If the person to be detected is not in a stationary state, it is determined that the person to be detected is not in a sleeping post state. At this time, the video image to be detected is continuously obtained from the monitoring video stream to judge the stationary state.

[0064] Step 130: Determine whether the person to be detected is in a sleeping state based on the sleeping confidence level.

[0065] Exemplarily, after obtaining the confidence of each head-shoulder area map of the person to be detected, each confidence of sleeping on the post can be compared with the confidence threshold, and the number of target confidences of sleeping on the post greater than the confidence threshold is determined. Based on the number, the proportion of the target confidences of sleeping on the post is determined. If the proportion is greater than the preset proportion threshold, it is determined that the person to be detected is in the state of sleeping on the post. If in the state of sleeping on the post, the sleeping behavior can be reported to the administrator terminal, and the sleeping prompt information can be output to the administrator terminal to correct the sleeping behavior of the person to be detected in time.

[0066] Exemplarily, after obtaining the sleeping confidence of each head and shoulder area diagram of the person to be detected, the average value and standard deviation of these sleeping confidence values ​​can be determined. If the average value is greater than a first threshold and the standard deviation is less than a second threshold, it is determined that the person to be detected is in a sleeping state.

[0067] The sleeping-on-the-job detection method provided by the embodiment of the present invention extracts the head-shoulder area map of the person to be detected from multiple frames of video images to be detected in the monitoring video stream, and uses the head-shoulder area map to first determine whether the person to be detected is in a stationary state. If the person is in a stationary state, a sleeping-on-the-job classification model is used to determine the sleeping-on-the-job confidence of each head-shoulder area map, and then based on the sleeping-on-the-job confidence, it is determined whether the person to be detected is in a sleeping-on-the-job state, thereby realizing sleeping-on-the-job detection of the person to be detected. Among them, the sleeping post classification model is obtained by performing comparative training between the initial sleeping post classification model and the pre-trained image feature extraction model based on the sample image pairs and the pre-trained image feature extraction model. The initial sleeping post classification model is obtained by replacing the first convolution layer of the pre-trained image feature extraction model with a deformable convolution layer. In this way, through the comparative training between the initial sleeping post classification model and the pre-trained image feature extraction model and combining the deformable convolution layer, the deformed image and the original image can be aligned, so that the sleeping post classification model can adapt to the stretching and deformation of the image, and improve the sleeping post classification model's ability to distinguish the posture details of the head and shoulder area of ​​the person to be detected, thereby improving the distinction between sleeping posts and non-sleeping posts, thereby reducing the false detection and missed detection of sleeping post detection, and improving the accuracy and recall rate of sleeping post detection.

[0068] based on Figure 1 In the sleeping post detection method of the corresponding embodiment, during the training process of the sleeping post classification model, considering the method of directly resampling the sample images to the same size for batch training, the images are stretched and deformed. This stretching deformation will reduce the distinction between the prone and head-down postures, making the postures of different labels similar, and there are problems such as difficulty in model optimization and unclear classification boundaries. Based on this, the embodiment of the present invention provides a joint training method based on deformable convolution, which performs comparative training between the initial sleeping post classification model and the pre-trained image feature extraction model, so that the trained sleeping post classification model can adapt to the deformation caused by image stretching, and uses a dynamic Loss function to adjust the optimization target of the model in stages.

[0069] Specifically, Figure 2 A flowchart of a training method for a sleep classification model provided by an embodiment of the present invention is shown as an example. Figure 2 As shown, the sleeping post classification model can be trained based on the following steps 210 to 250.

[0070] Step 210: Obtain a sample image pair.

[0071] The sample image pair includes a first sample image and a second sample image. The second sample image is obtained by resampling all images in the first sample image to the same size. The images in the first sample image and the images in the second sample image correspond to each other one by one.

[0072] For example, assuming that the first sample image includes image A1, image A2 and image A3, and resampling image A1, image A2 and image A3 to the same size can obtain image B1, image B2 and image B3. Then image A1 and image B1 constitute a sample image pair, image A2 and image B2 constitute a sample image pair, and image A3 and image B3 constitute a sample image pair, thereby obtaining the sample image pairs required for training.

[0073] The first sample image may include a first sample sleeping image and a first non-sample sleeping image, or may include a first sample sleeping head and shoulder region image and a first non-sample sleeping head and shoulder region image. Similarly, the second sample image may include a corresponding second sample sleeping image and a second non-sample sleeping image, or may include a corresponding second sample sleeping head and shoulder region image and a second non-sample sleeping head and shoulder region image.

[0074] Step 220: Extract a first image feature vector of the first sample image based on a pre-trained image feature extraction model.

[0075] Taking the first sample image including the first sample sleeping head and shoulder area image and the first non-sample sleeping head and shoulder area image as an example, the first sample sleeping head and shoulder area image and the first non-sample sleeping head and shoulder area image can be input into a pre-trained image feature extraction model, and image features can be extracted using the pre-trained image feature extraction model to obtain a first image feature vector output by the pre-trained image feature extraction model.

[0076] Step 230: extracting a second image feature vector of a second sample image corresponding to the first sample image based on the initial classification model, and determining a sample classification confidence level of the second sample image.

[0077] In the training process of the initial classification model, the first sample image input to the pre-trained image feature extraction model and the second sample image input to the initial classification model come from the same sample image pair, that is, the first sample image is the original image, and the second sample image is the image after the first sample image is resized. For example, the first sample image can be adjusted to the input size of the initial classification model using a bilinear interpolation method to obtain the second sample image.

[0078] For each sample image pair, the first sample image in the sample image pair can be input into a pre-trained image feature extraction model to obtain a first image feature vector output by the pre-trained image feature extraction model, and the second sample image in the sample image pair can be input into an initial image classification model to obtain a second image feature vector output by a global average pooling layer of the initial image classification model, as well as a sample pooling confidence of the second sample image output by a fully connected layer of the initial image classification model.

[0079] Step 240: Determine a loss function value based on the first image feature vector, the second image feature vector and the sample confidence level.

[0080] After obtaining the first image feature vector, the second image feature vector and the sample position confidence, the first image feature vector and the second image feature vector can be compared, and the loss function value can be determined based on the comparison result, the sample position confidence and the label information of the second sample image.

[0081] Specifically, in an example embodiment, determining the loss function value based on the first image feature vector, the second image feature vector and the sample confidence level may include: determining the relative entropy of the first image feature vector and the second image feature vector; determining the cross entropy of the second sample image based on the sample confidence level and the sample label value of the second sample image; determining the loss function value based on the relative entropy, the first weight of the relative entropy, the cross entropy and the second weight of the cross entropy; wherein the sum of the weight value of the first weight and the weight value of the second weight is 1.

[0082] Among them, relative entropy, also known as KL divergence (Kullback–Leibler divergence), can characterize the difference in probability distribution between the feature vector of the first image and the feature vector of the second image; cross entropy can characterize the loss between the sample confidence and the sample label value of the second sample image. In this way, by comparing the feature vector of the first image with the feature vector of the second image based on deformable convolution, the alignment of the second sample image with the first sample image can be achieved.

[0083] Exemplarily, a first weight may be added to the relative entropy, and a second weight may be added to the cross entropy, and the sum of the relative entropy after adding the first weight and the cross entropy after adding the second weight may be determined as the loss function value.

[0084] Specifically, the loss function value Loss can be determined according to the following formula (1):

[0085]

[0086] Among them, x i represents the confidence level of the sample, that is, the prediction probability during model training; i represents the sample label value; c represents the number of classification categories. For example, the classification categories may include sleeping posts and non-sleeping posts, then c = 2; m represents the dimension of the image feature vector; represents the second image feature vector, represents the first image feature vector; α is the first weight, and (1-α) is the second weight. It can be understood that is the cross entropy, This is the KL divergence.

[0087] Exemplarily, the weight value of the first weight α may be a preset fixed value, or may change with the change of training rounds, for example, may decrease with the increase of training rounds.

[0088] Specifically, the weight value of the first weight α can be determined based on the training rounds using the cosine annealing algorithm. For example, the weight value of the first weight α of each epoch process, that is, the weight value of the first weight α in each training process, can be determined according to the following formula (2):

[0089]

[0090] Among them, w represents the initial weight value; n represents the training round, that is, the nth round of training, and n∈[1,r]; r represents the total number of training times.

[0091] For example, the initial weight value w may be 0.8. In each subsequent round of training, the first weight α may be determined according to formula (2), and then the second weight (1-α) may be determined.

[0092] Figure 3 A schematic diagram showing the adjustment of the loss function by the first weight change during the training of the sleep classification model is shown in FIG. Figure 3 As shown, curve ① represents cross entropy, and curve ② represents KL divergence. Taking the total number of training times r = 80 as an example, the change of the first weight α can adjust the loss function Loss during the training of the sleep classification model into three stages:

[0093] In the first stage, the first weight of KL divergence is greater than the second weight of cross entropy, and the loss function Loss is mainly based on KL divergence. The optimization goal of the model is to reduce the gap between the feature vector of the deformed image (i.e., the second image feature vector) and the feature vector of the original size image (i.e., the first image feature vector), so as to adapt to the deformation of the image.

[0094] In the second stage, starting from the 34th round of training, the second weight of cross entropy is greater than the first weight of KL divergence. At this time, the model has a certain ability to adapt to image deformation, with the main goal of improving the prediction probability of the model output.

[0095] In the third stage, by the 80th round of training, the KL divergence is reduced to 0. At this time, the first weight of the KL divergence is no longer calculated, and the model continues to improve the prediction accuracy through fine-tuning.

[0096] Step 250: Adjust the model parameters of the initial sleeper classification model based on the loss function value until the loss function value is minimized to obtain the sleeper classification model.

[0097] For example, Figure 4The training principle diagram of the sleeper classification model in the embodiment of the present invention is exemplarily shown, referring to Figure 4 As shown in the figure, taking the ResNet18 network with ImageNet pre-trained weights as an example, the first convolution layer of the ResNet18 network can be replaced with a deformable convolution to form a deformable ResNet18 network, and the deformable ResNet18 network is used as the initial classification model for training.

[0098] For a pair of samples in the sample image pair, namely, the first sample image A1 as the original image and the second sample image A2 obtained after the first sample image A1 is resized, during the model training process, the first sample image A1 can be input into the ResNet18 network to obtain the first image feature vector output by the global average pooling layer of the ResNet18 network; at the same time, the second sample image A2 is input into the deformable ResNet18 network to obtain the second image feature vector output by the global average pooling layer of the deformable ResNet18 network and the predicted probability output by the fully connected layer. Then, the KL divergence is determined based on the first image feature vector and the second image feature vector, and the cross entropy is determined based on the predicted probability and the label information of the second sample image A2. The first weight α is added to the KL divergence and the second weight (1-α) is added to the cross entropy and then summed to obtain the loss function value Loss. The loss function value Loss can be compared with the preset loss threshold. If it is greater than the loss threshold, the model parameters of the deformable ResNet18 network are adjusted until the loss function value Loss is less than the preset loss threshold. In this case, the trained deformable ResNet18 network is the sleep-down classification model. Alternatively, the model parameters of the deformable ResNet18 network can be adjusted based on the loss function value Loss until the loss function value Loss is minimized. In this case, the trained deformable ResNet18 network is the sleep-down classification model.

[0099] The training method of the sleep-station classification model provided in this example embodiment, on the one hand, performs comparative training between the initial sleep-station classification model and the pre-trained image feature extraction model based on deformable convolution, which can realize the alignment of the resampled image and the original image in the sample image pair, adapt to the deformation caused by image stretching, and improve the resolution ability of the sleep-station classification model for posture details. On the other hand, during the model training process, the first weight α can be adjusted through the cosine annealing strategy to control the proportion of KL divergence and cross entropy at different stages. In the early stage, KL divergence is mainly used to control the model to adapt to the deformation caused by image stretching, and in the later stage, cross entropy loss is mainly used to improve the classification accuracy of the model, thereby further improving the adaptability of the sleep-station classification model to posture deformation.

[0100] Based on the above-mentioned embodiments of the detection method for sleeping on the post, considering that the sleeping on the post behavior is a continuous state of the person to be detected within a period of time, it is necessary to comprehensively classify the classification results of the person to be detected in multiple frames of video images to be detected. If the proportion of the classification results in the total number of frames is used to judge the sleeping state of the human body in this time period, it is easily affected by the fluctuation of the classification confidence. Based on this, the embodiment of the present invention determines whether the person to be detected is in the sleeping state through multi-frame fusion analysis on the basis of using the above-mentioned sleeping on the post classification model to determine the sleeping on the post confidence, and uses the above-mentioned sleeping on the post classification model The ability to resolve posture details is used to analyze the stability of the sleeping on the post confidence in multiple frames of video images to be detected, realizes the fusion of information of multiple frames of video images to be detected, and improves the robustness of the detection of sleeping on the post behavior.

[0101] Specifically, in an example embodiment, after determining the sleeping guard confidence of the head and shoulder area image in each frame of the video image to be detected by using the sleeping guard classification model, determining whether the person to be detected is in the sleeping guard state based on the sleeping guard confidence may include: determining the mean value and standard deviation of the sleeping guard confidence; when the mean value is greater than a first threshold and the standard deviation is less than a second threshold, determining that the person to be detected is in the sleeping guard state.

[0102] Among them, whether the average value is greater than the first threshold can be used to determine whether the sleeping classification result of the head and shoulder area map of the person to be detected in the multiple frames of video images to be detected is in the sleeping state, and whether the standard deviation is less than the second threshold can be used to determine whether the sleeping classification result of the head and shoulder area map of the person to be detected in the multiple frames of video images to be detected is stable.

[0103] For example, assuming that N frames of video images to be detected are acquired within a preset time period, the average value mean of the confidence levels of the video images to be detected corresponding to these frames can be determined according to the following formula (3):

[0104]

[0105] Among them, j represents the index of the frame to be detected video image, conf j Indicates the confidence level of the j-th frame of the video image to be detected.

[0106] Furthermore, the standard deviation std of the confidence levels of the video images to be detected corresponding to the images can be determined according to the following formula (4):

[0107]

[0108] According to the post sleeping detection method of this exemplary embodiment, taking the first threshold value of the average value as 0.7 and the second threshold value of the standard deviation as 0.05 as an example, the post sleeping detection method of the embodiment of the present invention can at least solve the following two problems of false alarm or missed alarm.

[0109] First, Figure 5 One of the schematic diagrams showing the results of the sleep guard test is shown in Figure 1. Figure 5 As shown in the figure, taking 5 frames of video images to be detected within a preset time period as an example, if the proportion of the number of frames with classification results of sleeping on the post to the total number of frames is used to judge the sleeping status of the person to be detected, assuming that its classification threshold is 0.7, and the confidence of sleeping on the post of at least 3 frames out of 5 frames is greater than the classification threshold, it is considered that the person to be detected is in the sleeping status. Figure 5 , the sleeping confidences corresponding to the 5 frames of video images to be detected fluctuate around 0.7, and only 2 of the 5 frames have sleeping confidences greater than the classification threshold. At this time, when the method of judging the sleeping state of the human body based on the proportion of the classification results in the total number of frames is used for judgment, it will be determined that the person to be detected is not in the sleeping state, resulting in missed detection. However, through calculation, it can be determined that the average sleeping confidence corresponding to the 5 frames is 0.704, and the standard deviation is 0.031. Using the sleeping detection method provided by the embodiment of the present invention, the average value is greater than 0.7 and the standard deviation is less than 0.05, so it can be determined that the person to be detected is in the sleeping state, which reduces the missed detection caused by considering only the single-frame judgment results and improves the recall rate of sleeping detection.

[0110] Second, Figure 6 The second diagram shows the results of the sleep-station test. Figure 6 As shown in the figure, taking 5 frames of video images to be detected within a preset time period as an example, when the sleeping confidence corresponding to multiple frames of video images to be detected fluctuates greatly, if the proportion of frames with classification results of sleeping on the post to the total number of frames is used to judge the sleeping state of the person to be detected, assuming that its classification threshold is 0.7, and at least 3 frames out of 5 frames have a sleeping confidence greater than the classification threshold, it is considered that the person to be detected is in the sleeping state. Figure 6 , the sleeping confidence of 4 frames in these 5 images is greater than the classification threshold, and it will be determined that the person to be detected is in a sleeping state. However, the large fluctuation of the sleeping confidence indicates that the sleeping classification model is unstable in predicting the sleeping state of the person to be detected or the person to be detected may not be in a static state during this period, and false detection may occur. In this case, using the sleeping detection method provided by the embodiment of the present invention, it can be determined that the average value of the sleeping confidence corresponding to these 5 frames of images is 0.816, which is greater than the first threshold of 0.7, and the standard deviation is 0.111, which is greater than the first threshold of 0.05. It is considered that the person to be detected is not in a sleeping state and will not be reported as sleeping. Subsequent video images to be detected can be obtained from the monitoring video stream, and sleeping detection can be continued. The occurrence of such false detection risks can be reduced through multi-frame fusion analysis, thereby improving the accuracy of sleeping detection.

[0111] The sleeping guard detection method provided by the embodiment of the present invention, based on the ability of the sleeping guard classification model to distinguish posture details, comprehensively considers the fluctuation of the sleeping guard confidence in multiple frames through fusion analysis of multiple frames of images, so as to judge whether the person to be detected is in a sleeping guard state. It can effectively reduce the influence of single-frame false detection and sleeping guard confidence fluctuation, and improve the robustness and accuracy of sleeping guard detection.

[0112] The joint training method of the sleeping guard classification model used in the sleeping guard detection method provided in the embodiment of the present invention and the method of judging the sleeping guard status by multi-frame fusion analysis based thereon can be applied to any backbone network model of image classification and have good scalability.

[0113] The following is a description of the post sleeping detection device provided by the present invention. The post sleeping detection device described below and the post sleeping detection method described above can be referenced to each other.

[0114] Figure 7 The structural diagram of the sleeping post detection device provided by the embodiment of the present invention is exemplarily shown. Figure 7 As shown, the post sleeping detection device may include: an image acquisition module 710, used to acquire at least two frames of video images to be detected in the monitoring video stream, and extract the head and shoulder area map of the person to be detected in each frame of the video image to be detected; an image classification module 720, used to determine the post sleeping confidence of each head and shoulder area map by using a post sleeping classification model; wherein the post sleeping classification model is obtained by performing comparative training between the initial post sleeping classification model and the pre-trained image feature extraction model based on the sample image pair and the pre-trained image feature extraction model, and the initial post sleeping classification model is obtained by replacing the first convolution layer of the pre-trained image feature extraction model with a deformable convolution layer; a post sleeping determination module 730, used to determine whether the person to be detected is in a post sleeping state based on the post sleeping confidence.

[0115] In an exemplary embodiment, the sleep guard detection device may also include a model training module, which may include: a sample acquisition unit, used to acquire a sample image pair, wherein the sample image pair includes a first sample image and a second sample image, the second sample image is obtained by resampling all images in the first sample image to the same size, and the first sample image and the second sample image correspond one to one; a first feature extraction unit, used to extract a first image feature vector of the first sample image based on a pre-trained image feature extraction model; a second feature extraction unit, used to extract a second image feature vector of the second sample image corresponding to the first sample image based on an initial sleep guard classification model, and determine the sample sleep guard confidence of the second sample image; a first determination unit, used to determine a loss function value based on the first image feature vector, the second image feature vector and the sample sleep guard confidence; an adjustment unit, used to adjust the model parameters of the initial sleep guard classification model based on the loss function value until the loss function value is minimized to obtain the sleep guard classification model.

[0116] In an exemplary embodiment, the first determination unit can be specifically used to: determine the relative entropy of the first image feature vector and the second image feature vector; determine the cross entropy of the second sample image based on the sample confidence level and the sample label value of the second sample image; determine the loss function value based on the relative entropy, the first weight of the relative entropy, the cross entropy and the second weight of the cross entropy; wherein the sum of the weight value of the first weight and the weight value of the second weight is 1.

[0117] In an exemplary embodiment, the weight value of the first weight decreases as the number of training rounds increases.

[0118] In an exemplary embodiment, the model training module further includes: a second determination unit, configured to determine a weight value of the first weight based on a training round using a cosine annealing algorithm.

[0119] In an exemplary embodiment, the sleeping post determination module 730 may include: a third determination unit, used to determine the mean value and standard deviation of the sleeping post confidence; and a fourth determination unit, used to determine that the person to be detected is in a sleeping post state when the mean value is greater than a first threshold and the standard deviation is less than a second threshold.

[0120] In an exemplary embodiment, the image classification module 720 may include: a fifth determination unit, used to determine the intersection-and-union ratio of two adjacent frames of head-and-shoulder area images; a sixth determination unit, used to determine that the person to be detected is in a stationary state when the intersection-and-union ratios corresponding to at least two frames of video images to be detected are greater than a third threshold; and a seventh determination unit, used to determine the sleeping position confidence of each head-and-shoulder area image by using the sleeping position classification model when it is determined that the person to be detected is in a stationary state.

[0121] Figure 8An example of a structural diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the sleep guard detection method provided by any of the above method embodiments.

[0122] Exemplarily, the processor 810 may include a central processing unit (CPU), a microprocessor, a network processor (NP), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0123] Exemplarily, the communication bus 840 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc.

[0124] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0125] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the sleeping post detection method provided by any of the above method embodiments.

[0126] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the sleeping post detection method provided by any of the above method embodiments.

[0127] By way of example, computer readable storage media include non-transitory computer readable storage media.

[0128] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting a sleeping post, characterized in that: include: Obtain at least two frames of video images to be detected in the monitoring video stream, and extract the head and shoulder area map of the person to be detected in each frame of the video image to be detected; Determine the confidence of each head-shoulder region image using a head-shoulder classification model; wherein the head-shoulder classification model is obtained by performing comparative training between the initial head-shoulder classification model and the pre-trained image feature extraction model based on the sample image pair and the pre-trained image feature extraction model, and the initial head-shoulder classification model is obtained by replacing the first convolution layer of the pre-trained image feature extraction model with a deformable convolution layer; Based on the sleeping-on-duty confidence level, it is determined whether the person to be detected is in a sleeping-on-duty state.

2. The method for detecting a sleeping position according to claim 1, wherein: The sleeper classification model is trained based on the following steps: Acquire the sample image pair, the sample image pair comprising a first sample image and a second sample image, the second sample image being obtained by resampling all images in the first sample image to the same size; the first sample image and the second sample image correspond one to one; Extracting a first image feature vector of the first sample image based on the pre-trained image feature extraction model; Based on the initial image classification model, extracting a second image feature vector of the second sample image corresponding to the first sample image, and determining a sample image confidence level of the second sample image; Determine a loss function value based on the first image feature vector, the second image feature vector and the sample confidence level; The model parameters of the initial sleep-station classification model are adjusted based on the loss function value until the loss function value is minimized to obtain the sleep-station classification model.

3. The method for detecting a sleeping position according to claim 2, wherein: The determining of the loss function value based on the first image feature vector, the second image feature vector and the sample confidence level includes: determining a relative entropy of the first image feature vector and the second image feature vector; Determine the cross entropy of the second sample image based on the sample confidence level and the sample label value of the second sample image; The loss function value is determined based on the relative entropy, the first weight of the relative entropy, the cross entropy and the second weight of the cross entropy; wherein the sum of the weight value of the first weight and the weight value of the second weight is 1.

4. The method for detecting a sleeping position according to claim 3, characterized in that: The weight value of the first weight decreases as the number of training rounds increases.

5. The method for detecting a sleeping position according to claim 4, characterized in that: Also includes: Based on the training rounds, a weight value of the first weight is determined using a cosine annealing algorithm.

6. The method for detecting a person sleeping on a post according to any one of claims 1 to 5, characterized in that: The determining whether the person to be detected is in a sleeping state based on the sleeping confidence level includes: Determining the mean and standard deviation of the confidence level of the sleep post; When the average value is greater than the first threshold value and the standard deviation is less than the second threshold value, it is determined that the person to be detected is in a sleeping state.

7. The method for detecting a person sleeping on the job according to claim 6, wherein: Determining the confidence level of each head and shoulder area graph using the head and shoulder classification model includes: Determine the intersection-and-union ratio of the head-shoulder region images of two adjacent frames; When the intersection-and-union ratios corresponding to the at least two frames of video images to be detected are all greater than a third threshold, it is determined that the person to be detected is in a stationary state; When it is determined that the person to be detected is in a stationary state, the sleeping position classification model is used to determine the sleeping position confidence level of each head-shoulder region image.

8. A device for detecting a person sleeping on the job, characterized in that: include: An image acquisition module is used to acquire at least two frames of video images to be detected in the monitoring video stream, and extract a head and shoulder area image of the person to be detected in each frame of the video image to be detected; An image classification module, used for determining the confidence of each head-shoulder region image by using a head-shoulder classification model; wherein the head-shoulder classification model is obtained by performing comparative training between the initial head-shoulder classification model and the pre-trained image feature extraction model on an initial head-shoulder classification model based on a sample image pair and a pre-trained image feature extraction model, and the initial head-shoulder classification model is obtained by replacing the first convolution layer of the pre-trained image feature extraction model with a deformable convolution layer; The sleeping-on-duty determination module is used to determine whether the person to be detected is in a sleeping-on-duty state based on the sleeping-on-duty confidence level.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the sleeping post detection method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the sleeping post detection method according to any one of claims 1 to 7 is implemented.

Images