Sleep post identification method and system

Abstract

The invention relates to the technical field of image recognition, in particular to a sleep post recognition method and system. The method comprises the following steps: collecting a sleep post video sample; extracting a sleep post picture sample from the sleep post video sample; labeling a label of each sleep post picture sample; determining a target detection area in the sleep post picture sample; initial feature extraction is carried out on the target detection area; determining a spatial attention feature of each initial feature based on a spatial attention mechanism; fusing the initial features and the spatial attention features to obtain new features; constructing a training sample by using the new features and the labels, and training by using the training sample to obtain a sleep post identification model; acquiring a to-be-detected video / picture in real time; whether the person in the to-be-detected video / picture is in the sleep state or not is judged by using the sleep recognition model. By adopting the method, the accuracy and accuracy of sleep recognition can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a sleeping post recognition method and system. Background Art

[0002] Sleeping on the job can easily lead to serious production accidents, and managers need to correct the behavior of employees sleeping on the job in a timely manner. Currently, most companies have achieved full coverage of surveillance videos, but if they rely solely on the naked eye to monitor, it will not only consume manpower but also be difficult to stay alert for a long time. With the development of deep learning, people can use algorithms to intelligently detect sleeping on the job, which can greatly improve efficiency while reducing the burden of manpower.

[0003] At present, the sleeping guard detection methods based on deep learning are mainly divided into two categories: target detection and human posture estimation. The target detection method uses the pictures of sleeping and non-sleeping people as a distinction, and directly detects these two states. Although it can achieve the purpose of sleeping guard recognition to a certain extent, it often leads to low recognition accuracy due to different sleeping postures, camera angles, etc., and cannot be further promoted. Human posture estimation recognizes by detecting the key points of the human body. When the human body is severely occluded, some key points cannot be detected, resulting in poor sleeping guard recognition based on human key points. Therefore, there are problems in the process of sleeping guard recognition that are poor in recognition effect and low classification accuracy, which need to be further solved. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a method and system for identifying a sleeping post, which improves the judgment precision and accuracy of a sleeping post identification model.

[0005] In order to achieve the above-mentioned purpose, the first aspect of an embodiment of the present invention provides a method for identifying a person who is sleeping on the post, the method comprising: collecting sleeping post video samples; extracting sleeping post picture samples from the sleeping post video samples; marking labels of each sleeping post picture sample; determining a target detection area in the sleeping post picture sample; performing initial feature extraction on the target detection area; determining the spatial attention features of each initial feature based on a spatial attention mechanism; fusing the initial features with the spatial attention features to obtain new features; constructing training samples using the new features and labels, and obtaining a sleeping post recognition model using the training samples; acquiring the video / picture to be detected in real time; and judging whether the person in the video / picture to be detected is in a sleeping post state using the sleeping post recognition model.

[0006] Based on the first aspect, in some embodiments of the present application, extracting the sleeping guard picture samples from the sleeping guard video samples includes: using opencv to extract frames from the sleeping guard video samples to obtain sleeping guard picture samples of various sleeping postures.

[0007] Based on the first aspect, in some embodiments of the present application, the target detection area is the head and shoulder area of ​​the human body; the determining of the target detection area in the sleeping post picture samples includes: marking the head and shoulder area of ​​the human body in some sleeping post picture samples to obtain a first training sample; using the first training sample to train a target detection model; determining the target detection area in the remaining sleeping post picture samples through the target detection model; the method also includes: determining the target detection area in the video / picture to be detected through the target detection model.

[0008] Based on the first aspect, in some embodiments of the present application, the method of obtaining the target detection model by training with the first training sample includes: obtaining the target detection model by training with the yolov5 algorithm.

[0009] Based on the first aspect, in some embodiments of the present application, the network structure of the yolov5 algorithm includes a Neck unit and a Predicition unit; wherein the Predicition unit includes: n Conv modules, used to respectively process feature maps of n scales output by the Neck unit, wherein the Conv module used to process the minimum scale feature map is the first Conv module; a CAG module, used to perform CAG processing on the feature map output by the first Conv module to obtain the weight coefficient of each channel in the feature map; the method also includes: multiplying the weight coefficient of each channel by the channel parameters of the feature map output by each Conv module.

[0010] Based on the first aspect, in some embodiments of the present application, the initial feature extraction for the target detection area includes: using a lightweight human posture feature extraction network to extract a human skeleton key point feature map from the target detection area.

[0011] Based on the first aspect, in some embodiments of the present application, if the sleeping post recognition model determines that the person in the video / picture to be detected is in a sleeping post state, the video / picture is stored and an alarm message is sent to the management personnel and / or corresponding personnel.

[0012] In the second aspect, the present application provides a sleeping post identification system, which includes: an acquisition module for acquiring videos / pictures to be detected in real time; a sleeping post identification model for determining whether a person in the video / picture to be detected is in a sleeping post state; and a sampling module for collecting sleeping post video samples; a sampling module for extracting sleeping post picture samples from the sleeping post video samples; a labeling module for labeling each sleeping post picture sample; a detection module for determining a target detection area in the sleeping post picture sample; a feature extraction module for performing initial feature extraction on the target detection area; a spatial attention module for determining the spatial attention features of each initial feature based on the spatial attention mechanism; a feature fusion module for fusing the initial features with the spatial attention features to obtain new features; and a model training module for constructing training samples using new features and labels, and training the training samples to obtain a sleeping post identification model.

[0013] Based on the second aspect, in some embodiments of the present application, the target detection area is the head and shoulder area of ​​the human body; the detection module includes: a target detection model, which is used to determine the target detection area in the sleeping post picture sample; and a model training unit, which is used to mark the head and shoulder area of ​​the human body in some sleeping post picture samples to obtain a first training sample, and use the first training sample to train the target detection model; the target detection model is also used to determine the target detection area in the video / picture to be detected. Based on the second aspect, in some embodiments of the present application, the system also includes: an alarm module, which is used to store the corresponding video / picture when the sleeping post recognition model detects that the person in the video / picture to be detected is in a sleeping post state, and send an alarm message to the management personnel and / or the corresponding personnel.

[0014] In a third aspect, the present application provides an electronic device comprising: at least one processor, at least one memory and a data bus; wherein the processor and the memory communicate with each other via the data bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the above-mentioned sleeping post identification method.

[0015] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores a computer program, wherein the computer program enables a computer to execute the above-mentioned sleeping post identification method.

[0016] The sleep-post identification method and system provided by the present application have at least the following beneficial effects:

[0017] 1) Through the cascade judgment of target detection and posture estimation, the area above the human body shoulder is processed in a targeted manner, the features are more concentrated, and the noise data interference in other areas is suppressed, so that the model can learn the features of this area more fully, and the model has better recognition accuracy under different angles and lighting conditions;

[0018] 2) The posture features and spatial attention features are integrated for classification and judgment, which makes the features richer and improves the classification accuracy.

[0019] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0021] Figure 1 It is a flow chart of the main steps of the sleeping post identification method provided in the embodiment of the present application;

[0022] Figure 2 is a schematic diagram of the network structure of the existing Prediction unit in the embodiment of the present application;

[0023] Figure 3 is a schematic diagram of the network structure of the improved Prediction unit in the embodiment of the present application;

[0024] Figure 4 Schematic diagram of the process of obtaining a key point map of a human skeleton in an embodiment of the present application;

[0025] Figure 5 It is a structural block diagram of an electronic device provided in an embodiment of the present application.

[0026] Description of Reference Numerals

[0027] 1-device; 101-processor; 102-memory; 103-bus. DETAILED DESCRIPTION

[0028] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0029] Example 1

[0030] This embodiment provides a method for identifying a sleeping post. Please refer to the flowchart of the main steps. Figure 1 As shown, the method specifically includes:

[0031] S1, collect sleeping post video samples;

[0032] S2, extracting sleeping guard picture samples from sleeping guard video samples;

[0033] The collected video samples of sleeping guards include daily surveillance videos and artificially simulated videos of sleeping guard behaviors. We select sleeping guard clips from them and use opencv to extract frames to obtain enough pictures to ensure that all sleeping guard postures are covered as much as possible.

[0034] S3, label each sleeping post image sample;

[0035] The label here specifically gives the conclusion as to whether the staff in the sleeping duty video sample or the sleeping duty picture sample is in a sleeping duty state.

[0036] S4, determining the target detection area in the sleeping guard image sample;

[0037] Specifically, the target detection area in this embodiment mainly refers to the head and shoulder area of ​​the human body in the picture / video.

[0038] For example, the target detection area in the sleeping guard picture sample (or the video / picture to be detected) can be determined by training the target area detection model. Specifically, the model training process is as follows:

[0039] The head and shoulder regions of the human body in some samples of sleeping photos are labeled to obtain a first training sample; and the target detection model is trained using the first training sample.

[0040] Finally, the target detection model can be used to determine the target detection area in the remaining sleeping guard image samples, which can speed up the training of the sleeping guard recognition model and quickly obtain training samples. In addition, the model can also be used to identify the target detection area of ​​the video / image to be detected during the use stage.

[0041] S5, performing initial feature extraction on the target detection area;

[0042] Specifically, lightweight openpose (such as lightweight network MobileNet-V1) can be used to extract the posture of the target area above the shoulder, obtain the skeleton key point image, and draw the lines between the skeleton nodes in color. 5,000 images are collected as the training set for the subsequent classification model.

[0043] In this embodiment, a lightweight network MobileNet-V is used. Compared with a heavyweight network (such as VGG-16), its parameter volume is only about 3% of the original one, and the reasoning speed is greatly improved. Figure 2As shown in the figure, the feature map outputs the human skeleton key point map after the initial stage and refinement stage.

[0044] S6. Determine the spatial attention features of each initial feature based on the spatial attention mechanism;

[0045] S7, fusing the initial features with the spatial attention features to obtain new features;

[0046] The spatial attention module is introduced in this embodiment to make the model pay more attention to meaningful features. Specifically, the spatial attention feature in this embodiment refers to the weight coefficient of each region in the image.

[0047] Exemplarily, after passing through the above-mentioned openpose feature extraction network, a feature F (initial feature) of H×W×C is obtained. First, the maximum pooling and average pooling of a channel dimension are performed to obtain two H×W×1 channel descriptions, and these two descriptions are spliced ​​together according to the channel to form a H×W×2 matrix. Then, after passing through a 7×7 convolution layer, the activation function is Sigmoid, and the weight coefficient Ms (spatial attention feature) is obtained. Among them, the weight coefficient has the same size as the feature F. By multiplying the weight coefficient and the feature F element by element, the fusion of F and the spatial attention feature can be realized, and then a new feature that integrates spatial attention can be obtained.

[0048] S8, constructing training samples using new features and labels, and using the training samples to train a sleeping post recognition model;

[0049] Specifically, the sleeping post recognition model can adopt the MobileNet-V3 binary classification network model.

[0050] S9, real-time acquisition of the video / picture to be detected;

[0051] S10: Using the sleeping post recognition model, determine whether the person in the video / image to be detected is in a sleeping post state.

[0052] Preferably, if the sleeping post recognition model determines that the person in the video / picture to be detected is in a sleeping post state, the video / picture is stored and an alarm message is sent to the management personnel and / or corresponding personnel.

[0053] For some positions, sleeping on the job can easily lead to serious production accidents, and managers need to correct the sleeping behavior of on-the-job personnel in a timely manner. Therefore, the sleeping recognition model trained in this implementation is mainly used to identify the real-time acquired videos / pictures to be detected.

[0054] Example 2

[0055] This embodiment specifically describes the target detection model in step S4 in Embodiment 1.

[0056] Different from object detection through the entire human body, this method uses labeled data to train an object detection model that only recognizes the area above the human shoulders. Since the labeled data above the human shoulders is used as the training data set, the object detection model can accurately recognize the area above the human shoulders in the image. Judging only by the features of the head and shoulder area allows the model to learn the features of this area more focused, reducing the influence of noise data that contributes less to the judgment of sleeping behavior, avoiding the problem of poor model recognition when the target is sideways or severely occluded, and improving the robustness of the model.

[0057] Specifically, the target detection model can be trained using the yolov5 algorithm to obtain a target detection model.

[0058] Furthermore, the network structure of the YOLOv5 algorithm used in this embodiment mainly consists of four parts: input end, Backbone, Neck, and Predicition. Except for the Prediction part, the other parts adopt the official recommended structure. Prediction is different from the structure of YOLOv5 that directly predicts (the network structure of the conventional Prediction unit is as follows Figure 3 As shown), in this embodiment, a channel attention guidance module CAG based on a channel attention mechanism is adopted (the network structure of the Prediction unit in this application is as shown Figure 4 As shown in Figure 2, this alleviates the aliasing effect caused by FPN fusion.

[0059] Specifically, the output of the Neck structure of yolov5 contains three levels of features (i.e., feature maps of three scales, and the corresponding Predicition unit includes three Conv modules for processing the feature maps of the three scales respectively). The feature map of the smallest scale contains more semantic information. Therefore, only using the feature map of the smallest scale for CAG processing can achieve the purpose of making full use of semantic information, while also avoiding excessive model parameters.

[0060] like Figure 4As shown in the figure, after the CAG module receives the data, one branch performs global average pooling (Global AvgPool) and the other branch performs global maximum pooling (Global Max Pool), respectively obtaining a 1*1*C (C represents the number of channels, which should be 255 here) matrix, and then passes through the fully connected layer, element-by-element addition and sigmoid processing to obtain a 1*1*255 matrix. The value of each element in the matrix is ​​between 0 and 1, representing the weight coefficient of each channel. After obtaining the weight coefficient of each channel through CAG, it is multiplied by the above-mentioned three-level features to obtain matrices of 80*80*255, 40*40*255 and 20*20*255 respectively. The subsequent detection and classification steps are the same as the open source yolov5.

[0061] Example 3

[0062] The present application provides a system for identifying people sleeping on the post, and the system includes: an acquisition module, which is used to acquire videos / pictures to be detected in real time; a sleeping post identification model, which is used to determine whether a person in the video / picture to be detected is in a sleeping post state; and a sampling module, which is used to collect sleeping post video samples; a sampling module, which is used to extract sleeping post picture samples from the sleeping post video samples; a labeling module, which is used to label each sleeping post picture sample; a detection module, which is used to determine a target detection area in the sleeping post picture sample; a feature extraction module, which is used to perform initial feature extraction on the target detection area; a spatial attention module, which is used to determine the spatial attention features of each initial feature based on the spatial attention mechanism; a feature fusion module, which is used to fuse the initial features with the spatial attention features to obtain new features; and a model training module, which is used to construct training samples using the new features and labels, and train the training samples to obtain a sleeping post identification model.

[0063] Furthermore, the target detection area is the head and shoulder area of ​​the human body; the detection module includes: a target detection model, used to determine the target detection area in the sleeping post picture sample; and a model training unit, used to mark the head and shoulder area of ​​the human body in some sleeping post picture samples to obtain a first training sample, and use the first training sample to train the target detection model; the target detection model is also used to determine the target detection area in the video / picture to be detected.

[0064] Furthermore, the system also includes: an alarm module, which is used to store the corresponding video / picture and send an alarm message to the management personnel and / or corresponding personnel when the sleeping post recognition model detects that the person in the video / picture to be detected is in a sleeping post state.

[0065] The above-mentioned sleeping guard identification system also includes a processor and a memory. The above-mentioned acquisition module, sleeping guard identification model, sampling module, sampling module and detection module are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.

[0066] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the detection accuracy can be improved by adjusting the kernel parameters.

[0067] An embodiment of the present invention provides a storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned sleeping post identification method is implemented.

[0068] An embodiment of the present invention further provides a processor, which is used to run a program, wherein the program executes a sleeping post identification method when running.

[0069] The embodiment of the present invention further provides an electronic device 1, such as Figure 5 As shown, the device 1 includes: at least one processor 101, at least one memory 102 and a data bus 103; wherein the processor 101 and the memory 102 communicate with each other through the data bus 103; the memory 102 stores program instructions that can be executed by the processor, and the processor 101 calls the program instructions to execute the above method. The device 1 in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0070] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: collecting sleeping post video samples; extracting sleeping post picture samples from the sleeping post video samples; marking labels for each sleeping post picture sample; determining a target detection area in the sleeping post picture sample; performing initial feature extraction on the target detection area; determining the spatial attention features of each initial feature based on the spatial attention mechanism; fusing the initial features with the spatial attention features to obtain new features; constructing training samples using the new features and labels, and obtaining a sleeping post recognition model using the training samples; acquiring the video / picture to be detected in real time; and determining whether the person in the video / picture to be detected is in a sleeping post state using the sleeping post recognition model.

[0071] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] In a typical configuration, the device includes one or more processors (CPU), memory and bus. The device may also include input / output interface, network interface and the like.

[0073] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.

[0074] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0075] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0076] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0077] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A sleeping post identification method, characterized in that: The method comprises: Collect samples of sleeping posts videos; Extract sleeping post picture samples from sleeping post video samples; Label the sample images of each sleeping post; Determine the target detection area in the sleeping guard image sample; Performing initial feature extraction on the target detection area; Determine the spatial attention features of each initial feature based on the spatial attention mechanism; Fusion of initial features and spatial attention features to obtain new features; Use the new features and labels to construct training samples, and use the training samples to train a sleeping post recognition model; Obtain the video / picture to be detected in real time; The sleeping guard recognition model is used to determine whether the person in the video / image to be detected is in a sleeping guard state.

2. The sleeping post identification method according to claim 1, characterized in that: The step of extracting the sleeping guard picture sample from the sleeping guard video sample comprises: Use OpenCV to extract frames from the sleeping guard video samples to obtain sleeping guard picture samples of various sleeping guard postures.

3. The sleeping post identification method according to claim 1, characterized in that: The target detection area is the head and shoulder area of ​​the human body; Determining the target detection area in the sleeping guard picture sample includes: Label the head and shoulder areas of the human body in some sleeping guard image samples to obtain the first training sample; Using the first training sample to train a target detection model; Determine the target detection area in the remaining sleeping guard image samples through the target detection model; The method further comprises: The target detection model is used to determine the target detection area in the video / image to be detected.

4. The sleeping post identification method according to claim 3, characterized in that: The method of obtaining a target detection model by training the first training sample includes: The target detection model is trained using the yolov5 algorithm.

5. The sleeping post identification method according to claim 4, characterized in that: The network structure of the yolov5 algorithm includes a Neck unit and a Predicition unit; Wherein, the Predicition unit includes: n Conv modules, used to process the feature maps of n scales output by the Neck unit respectively, wherein the Conv module used to process the feature map of the smallest scale is the first Conv module; A CAG module, used to perform CAG processing on the feature map output by the first Conv module to obtain the weight coefficient of each channel in the feature map; The method further comprises: The weight coefficients of the channels are multiplied by the channel parameters of the feature maps output by each Conv module.

6. The sleeping post identification method according to claim 1, characterized in that: The initial feature extraction for the target detection area includes: A lightweight human posture feature extraction network is used to extract the human skeleton key point feature map from the target detection area.

7. The sleeping post identification method according to claim 1, characterized in that: If the sleeping post recognition model determines that the person in the video / picture to be detected is in a sleeping post state, the video / picture will be stored and an alarm message will be sent to the management personnel and / or corresponding personnel.

8. A sleeping post identification system, characterized in that: The system comprises: Acquisition module, used to acquire the video / picture to be detected in real time; The sleeping guard recognition model is used to determine whether the person in the video / image to be detected is sleeping on the guard; and Sampling module, used to collect sleeping post video samples; A sampling module is used to extract sleeping guard picture samples from sleeping guard video samples; The labeling module is used to label the samples of each sleeping post image; A detection module, used to determine the target detection area in the sleeping guard image sample; A feature extraction module, used for performing initial feature extraction on the target detection area; A spatial attention module, used to determine the spatial attention features of each initial feature based on the spatial attention mechanism; Feature fusion module, used to fuse the initial features with the spatial attention features to obtain new features; The model training module is used to construct training samples using new features and labels, and use the training samples to train a sleeping post recognition model.

9. An electronic device, characterized in that: include: At least one processor, at least one memory and a data bus; wherein the processor and the memory communicate with each other via the data bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the sleeping post identification method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores a computer program, and the computer program enables a computer to execute the sleeping post identification method according to any one of claims 1 to 7.

Images