Sleep behavior detection method and device

Abstract

The invention provides a sleep behavior detection method and apparatus. The method comprises the steps of obtaining a to-be-detected image within a target time; inputting the to-be-detected image in the target time into the sleeping posture detection model to obtain a sleeping posture detection result output by the sleeping posture detection model; wherein the sleeping posture detection model is obtained by training based on a training image data set and a detection label corresponding to the training image data set, and the sleeping posture detection model is used for performing feature fusion based on image features extracted from an input to-be-detected image within target time and performing target detection on the fused features to obtain a sleeping posture detection result; tracking the person according to the sleeping posture detection result to obtain a person tracking result; and according to the sleeping posture detection result and the person tracking result, determining a sleeping post behavior detection result corresponding to each person. According to the invention, sleeping posture detection is carried out on the to-be-detected image in the target time to track the person, so that whether the person has a sleeping behavior is comprehensively evaluated, and the possibility of misjudgment is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and device for detecting post sleeping behavior. Background Art

[0002] In many work environments, especially those that require high alertness, such as security, traffic monitoring, and medical monitoring, keeping workers awake is the key to ensuring safety and efficiency. However, long hours of work and a monotonous environment can easily lead to fatigue and dozing off, which not only reduces work efficiency but can also pose serious safety risks. Therefore, it is important to detect on-site workers' sleeping behavior.

[0003] Traditional monitoring methods include manual supervision, monitoring fatigue status using biometrics such as heart rate and eye movement, video monitoring, and manual observation. However, these methods either require wearing special equipment, which provides a poor user experience, or rely on manual analysis, which is inefficient and may result in missed detections. Summary of the invention

[0004] The invention provides a method and device for detecting post sleeping behavior, which are used to solve the defects of low efficiency and possible missed detection in the prior art of post sleeping behavior detection, realize continuous monitoring of personnel, and improve the accuracy of post sleeping behavior detection.

[0005] The present invention provides a method for detecting sleeping-on-duty behavior, comprising: acquiring images to be detected within a target time; inputting the images to be detected within the target time into a sleeping position detection model, and obtaining sleeping position detection results output by the sleeping position detection model; wherein the sleeping position detection model is trained based on a training image data set and detection labels corresponding to the training image data set, and the sleeping position detection model is used to perform feature fusion based on image features extracted from the input images to be detected within the target time, and perform target detection on the fused features to obtain sleeping position detection results; according to the sleeping position detection results, tracking personnel to obtain personnel tracking results; and determining sleeping-on-duty behavior detection results corresponding to each person according to the sleeping position detection results and the personnel tracking results.

[0006] According to a method for detecting post sleeping behavior provided by the present invention, a sleeping posture detection model includes: a feature extraction layer, which extracts features of images to be detected within an input target time to obtain image features corresponding to each frame of the image; a feature fusion layer, which fuses corresponding image features for each frame of the image to be detected to obtain fused features corresponding to each frame of the image to be detected; and a target detection layer, which performs sleeping posture detection on the fused features corresponding to each frame of the image to be detected, and uses non-maximum suppression to obtain the corresponding sleeping posture detection result.

[0007] According to a method for detecting post sleeping behavior provided by the present invention, before inputting the images to be detected within a target time into a sleeping posture detection model, the method comprises: obtaining a training image data set, the training image data set comprising a plurality of training image frame sequences; obtaining a detection label for each frame of training image corresponding to each image frame sequence, the detection label comprising a sleeping posture category label, a person position label, and a target situation label of whether the person position determined according to the person position label contains the corresponding person; for each iterative training, selecting a preset number of image frame sequences from the training image data set, and respectively inputting the selected image frame sequences into the model to be trained, to obtain a sleeping posture prediction result of the corresponding image frame sequence output by the model to be trained, the sleeping posture prediction result comprising a sleeping posture detection result, a person detection frame label, and a target situation label of whether the corresponding person is contained in the person position determined according to the person position label; and the confidence that the corresponding person is included in the person detection box; construct a category loss function according to the sleeping position detection result and the corresponding sleeping position category label, construct a position loss function according to the person detection box and the corresponding person position label, and construct a confidence loss function according to the confidence that the corresponding person is included in the person detection box and the corresponding target situation label; obtain the total loss function according to the category loss function, the position loss function and the confidence loss function, and determine the gradient information of the total loss function based on the back propagation algorithm; according to the gradient information of the total loss function, use the preset optimization algorithm to update the network parameters of the model to be trained, and perform the next iterative training until the number of iterations meets the preset training number or the corresponding total loss function converges, and a trained sleeping position detection model is obtained.

[0008] According to a method for detecting sleeping on the job behavior provided by the present invention, the sleeping posture detection result includes a target detection frame of each frame of the image to be detected within a target time. According to the sleeping posture detection result, the person is tracked to obtain a person tracking result, including: obtaining intersection-over-union (IoU) information according to the target detection frames of two adjacent frames of the image to be detected; and evaluating the matching degree of the target detection frames of two adjacent frames of the image to be detected according to the IoU information to determine whether they are the same person target, thereby obtaining a person tracking result.

[0009] According to a method for detecting post sleeping behavior provided by the present invention, a sleeping posture detection result includes a sleeping posture recognition result of each frame of an image to be detected within a target time, and a sleeping posture behavior detection result corresponding to each person is determined according to the sleeping posture detection result and the person tracking result, including: obtaining the sleeping posture recognition result of each person target in each frame of the image to be detected according to the sleeping posture detection result and the person tracking result; determining the number of results of the sleeping posture recognition result of each person being a sleeping category according to the sleeping posture recognition result of each person target in all frames of the image to be detected within the target time; comparing the number of results with a preset post sleeping threshold, and determining that the person status corresponding to the target time is a sleeping state based on the number of results being greater than the preset post sleeping threshold; otherwise, it is a non-sleeping state; obtaining the sleeping behavior detection result of the corresponding person according to the person status corresponding to the target time and the historical status score obtained previously; wherein the historical status score is obtained based on the cumulative score of the person status determined at the historical target time prior to the target time.

[0010] According to a method for detecting post sleeping behavior provided by the present invention, the historical status score includes a post sleeping behavior score and an interruption timing score, and the post sleeping behavior detection result of the corresponding personnel is obtained according to the personnel status corresponding to the target time and the historical status score obtained in advance, including: if the personnel status corresponding to the target time is the post sleeping state, the post sleeping behavior of the corresponding personnel is timed, and the post sleeping behavior score of the corresponding personnel is increased by one point and the interruption timing score is reduced by one point; otherwise, the post sleeping behavior of the corresponding personnel is interrupted, and the interruption timing score of the corresponding personnel is increased by one point and the post sleeping behavior score is reduced by one point, so as to obtain the post sleeping behavior detection result of the corresponding personnel.

[0011] According to a method for detecting post-sleeping behavior provided by the present invention, after obtaining the detection result of the corresponding person's post-sleeping behavior, the method comprises: performing violation handling on the corresponding person according to the detection result of the corresponding person's post-sleeping behavior in combination with a preset violation handling strategy; wherein the preset violation handling strategy is used to limit the violation handling measures corresponding to the detection result of the post-sleeping behavior.

[0012] The present invention also provides a device for detecting post sleeping behavior, comprising: an image acquisition module for acquiring an image to be detected within a target time; The sleeping posture detection module inputs the images to be detected within the target time into the sleeping posture detection model, and obtains the sleeping posture detection results output by the sleeping posture detection model; wherein the sleeping posture detection model is trained based on the training image data set and the detection labels corresponding to the training image data set, and the sleeping posture detection model is used to perform feature fusion based on the image features extracted from the input images to be detected within the target time, and perform target detection on the fused features to obtain the sleeping posture detection results; the personnel tracking module tracks the personnel according to the sleeping posture detection results to obtain the personnel tracking results; the behavior monitoring module determines the sleeping behavior detection results corresponding to each personnel according to the sleeping posture detection results and the personnel tracking results.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned methods for detecting post sleeping behavior is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned methods for detecting post sleeping behavior is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, any of the above-mentioned methods for detecting post sleeping behavior is implemented.

[0016] The method and device for detecting post sleeping behavior provided by the present invention perform sleeping posture detection on images to be detected within a target time through a sleeping posture detection model to avoid misjudgment caused by single frame image detection, thereby more accurately judging whether a person is in a sleeping state and improving the accuracy of detection. The sleeping posture detection result is further combined with the person tracking result to track the person and realize continuous monitoring of the person's status, thereby determining the corresponding sleeping posture detection result of each person according to the sleeping posture detection result and the person tracking result, comprehensively evaluating whether the person has sleeping posture behavior and reducing the possibility of misjudgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is one of the flow diagrams of the method for detecting the post sleeping behavior provided by the present invention; Figure 2 This is the second flow chart of the method for detecting the post sleeping behavior provided by the present invention; Figure 3 It is a structural schematic diagram of a device for detecting post sleeping behavior provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Figure 1 Schematic diagram of the process of the method for detecting the behavior of sleeping on the post provided by the present invention. Figure 1 As shown, the method comprises: S11, acquiring the image to be detected within the target time; S12, inputting the image to be detected within the target time into the sleeping position detection model, and obtaining the sleeping position detection result output by the sleeping position detection model; wherein the sleeping position detection model is trained based on the training image data set and the detection labels corresponding to the training image data set, and the sleeping position detection model is used to perform feature fusion based on the image features extracted from the input image to be detected within the target time, and perform target detection on the fused features to obtain the sleeping position detection result; S13, tracking the person according to the sleeping posture detection result to obtain a person tracking result; S14, determining the corresponding sleeping behavior detection result of each person according to the sleeping posture detection result and the personnel tracking result.

[0021] It should be noted that the step numbers "S1N" in this manual do not represent the order of the method for detecting the behavior of sleeping on the post. Figure 2 The present invention describes the method for detecting the post sleeping behavior.

[0022] Step S11, obtaining the image to be detected within the target time.

[0023] In this embodiment, obtaining the image to be detected within the target time includes: based on a preset shooting interval, shooting the target monitoring area to obtain the corresponding image to be detected and the corresponding shooting time; according to the image to be detected and the corresponding shooting time; determining the shooting duration according to the shooting time and the preset shooting interval, and based on the shooting duration being consistent with the target time, using the corresponding image to be detected as the image to be detected within the target time.

[0024] It should be noted that the preset shooting interval can be set according to the target time and the preset number of images. For example, if the target time is one minute and the preset number of images is 60, the preset shooting interval is 1s. In addition, the monitored area can be monitored in real time by the camera equipment to remind the on-site staff to take timely measures, thereby greatly reducing the incidence of safety accidents and greatly ensuring the safety of on-site personnel and equipment.

[0025] Furthermore, when shooting the target monitoring area based on the preset shooting interval, the shooting angle can be adjusted based on the preset adjustment frequency using a spherical camera or other camera equipment, or different fixed viewing angles can be selected based on the preset adjustment frequency to shoot, thereby obtaining images to be detected at different viewing angles. It should be noted that the preset adjustment frequency can be configured according to the image of the actual required viewing angle. For example, if the viewing angles of the images to be detected of adjacent frames are different, the preset adjustment frequency is 1s, which is not further limited here.

[0026] In addition, when photographing the target monitoring area, it also includes: photographing the target detection area using at least two camera devices with different shooting angles to obtain corresponding images to be detected.

[0027] Step S12, input the image to be detected within the target time into the sleeping position detection model, and obtain the sleeping position detection result output by the sleeping position detection model; wherein the sleeping position detection model is trained based on the training image data set and the detection labels corresponding to the training image data set, and the sleeping position detection model is used to perform feature fusion based on the image features extracted from the input image to be detected within the target time, and perform target detection on the fused features to obtain the sleeping position detection result.

[0028] In this embodiment, the viewing angles of two adjacent frames of images to be detected within the target time are different. The sleeping position detection model includes: a feature extraction layer, which performs feature extraction on the input images to be detected within the target time to obtain image features corresponding to each frame of the image; a feature fusion layer, which performs feature fusion on the corresponding image features for each frame of the image to be detected to obtain fused features corresponding to each frame of the image to be detected; a target detection layer, which performs sleeping position detection on the fused features corresponding to each frame of the image to be detected, and uses non-maximum suppression to obtain sleeping position detection results.

[0029] It should be noted that the sleeping posture detection result includes the sleeping posture recognition result and target detection frame corresponding to each frame of the image to be detected within the target time. The target detection frame is used to indicate the position of the corresponding person target, and the sleeping posture recognition result is used to indicate whether the person in the corresponding target detection frame is in a sleeping state.

[0030] Correspondingly, the images to be detected within the target time are input into the sleeping position detection model to obtain the sleeping position detection results output by the sleeping position detection model, including: inputting the images to be detected within the target time into the feature extraction layer for feature extraction, and obtaining the image features corresponding to each frame of images output by the feature extraction layer; inputting the image features corresponding to each frame of images into the feature fusion layer, so as to perform feature fusion on the corresponding image features for each frame of images to be detected, and obtain the fusion features corresponding to each frame of images to be detected output by the feature fusion layer; inputting the fusion features corresponding to each frame of images to be detected into the target detection layer for sleeping position detection, and obtaining the sleeping position detection results output by the target detection layer.

[0031] In an optional embodiment, before inputting the images to be detected within the target time into the sleeping position detection model, the method includes: obtaining a training image data set, the training image data set including a plurality of training image frame sequences; obtaining a detection label for each frame of the training image corresponding to each image frame sequence, the detection label including a sleeping position category label, a person position label, and a target situation label of whether the person position determined according to the person position label contains the corresponding person; for each iterative training, selecting a preset number of image frame sequences from the training image data set, and inputting the selected image frame sequences into the model to be trained respectively, to obtain the sleeping position prediction results of the corresponding image frame sequences output by the model to be trained, the sleeping position prediction results including the sleeping position detection results, the person detection frame and the person detection frame. The box contains the confidence of the corresponding person; according to the sleeping posture detection result and the corresponding sleeping posture category label, a category loss function is constructed, according to the person detection box and the corresponding person position label, a position loss function is constructed, and according to the confidence of the corresponding person contained in the person detection box and the corresponding target situation label, a confidence loss function is constructed; according to the category loss function, the position loss function and the confidence loss function, the total loss function is obtained, and based on the back propagation algorithm, the gradient information of the total loss function is determined; according to the gradient information of the total loss function, the preset optimization algorithm is used to update the network parameters of the model to be trained, and the next iterative training is performed until the number of iterations meets the preset training number or the corresponding total loss function converges, so as to obtain a trained sleeping posture detection model.

[0032] It should be added that the preset optimization algorithm can be configured according to actual needs. For example, the preset optimization algorithm can be stochastic gradient descent (SGD), Adam, etc., which is not further limited here.

[0033] Specifically, obtaining a training image data set includes: based on a preset time interval, using multiple shooting devices to shoot the monitoring area, and based on the shooting time being consistent with the target time, obtaining a target number of image frame sequences; and constructing a training image data set based on the target number of image frame sequences. In addition, when shooting the monitoring area, it can also be obtained based on different attitude angles, obstructions, lighting, weather and other factors to improve the robustness of subsequent model training.

[0034] Furthermore, after photographing the monitoring area using a plurality of photographing devices, the method further includes: preprocessing the photographed images, wherein the preprocessing includes at least one of denoising and color correction.

[0035] In order to avoid redundancy between consecutive frame training images, after using multiple shooting devices to shoot the monitoring area, it also includes: based on the preset time length, dividing the training image frame sequence shot based on the preset time interval to obtain different groups; identifying the training images in each group to eliminate images that do not contain human targets.

[0036] In addition, after obtaining the training image data set, the following steps are performed: marking the person position of each training image in the training image data set, the sleeping position of the corresponding person, and marking whether the corresponding person is included in the corresponding person position, so as to obtain the sleeping position category label, the person position label, and the target situation label.

[0037] In addition, after obtaining the training image data set, it also includes: using a data enhancement strategy to perform data enhancement on the training images in the training image data set. Specifically, the data enhancement strategy includes at least one of flipping, rotating, cropping, deforming and scaling; and / or, the data enhancement strategy includes at least one of noise, blurring, color conversion, erasing and filling. It should be noted that the data enhancement strategy selected based on this embodiment is suitable for data enhancement of the above-mentioned training images, so as to increase the amount of training image data, and is beneficial to the subsequent model training process, greatly improving the model's sleeping posture detection accuracy in scenes such as lighting, occlusion, incompleteness, and large deflection angles.

[0038] When actually selecting a data augmentation strategy, you can choose any one of flipping, rotating, cropping, deforming or scaling, or you can choose any one of noise, blurring, color change, erasing and filling, or you can choose at least two of flipping, rotating, cropping, deforming and scaling; or you can choose at least two of noise, blurring, color change, erasing and filling; or you can choose at least two of flipping, rotating, cropping, deforming, scaling, noise, blurring, color change, erasing and filling.

[0039] It should be noted that the network to be trained can be an existing network built into the training device, which usually includes a network structure, or can be other networks specified by the user, such as the target detection network YOLOv7, etc. The network to be trained usually includes a feature extraction layer for extracting corresponding image features, a feature fusion layer for fusing features, a target detection layer for detecting fused features, and a loss function; according to the preset iteration rules, the above training image data set or the training image data set after data enhancement is input into the model to be trained for training, and a trained sleeping posture detection model is obtained.

[0040] In an optional embodiment, before constructing a category loss function based on the sleeping posture detection results and the corresponding sleeping posture category labels, constructing a position loss function based on the person detection frame and the corresponding person position label, and constructing a confidence loss function based on the confidence that the corresponding person is included in the person detection frame and the corresponding target situation label, it includes: assigning a detection label to the sleeping posture prediction result based on a preset label assignment strategy.

[0041] Specifically, the sleeping posture prediction result includes the sleeping posture prediction result of each training image in the training image frame sequence; based on the preset label allocation strategy, the detection label is allocated to the sleeping posture prediction result, including: according to each training image in the training image frame sequence corresponding to the sleeping posture prediction result, each training image is grid-divided to obtain a corresponding grid unit; based on the preset person size range, preset position and the preset position change range of the person between adjacent frames, the sleeping posture category and the number of people contained in each grid unit of each frame of the training image are estimated; according to the estimated target category and number contained in each grid unit of each frame of the training image, the candidate target area of ​​each frame of the training image is determined, and the candidate target area is composed of grid units containing targets determined based on the sleeping posture category and the number of people contained in each grid unit; for each candidate target area, the intersection and union ratio of the candidate target area and the sleeping posture category label is determined, and based on the first preset threshold, the candidate target area that is not greater than the first preset threshold is screened out to obtain a coarse matching area; based on the target semantic features and context information obtained by the auxiliary branch contained in the YOLOv7 network structure, combined with the preset data feature matching conditions, the matching degree between the coarse matching area and the sleeping posture category label is evaluated to obtain the label allocation result.

[0042] Step S13, tracking the person according to the sleeping posture detection result to obtain the person tracking result.

[0043] In this embodiment, a person is tracked based on the sleeping posture detection result to obtain a person tracking result, including: obtaining intersection-over-union (IoU) information based on the target detection frames of two adjacent frames of images to be detected; based on the IoU information, evaluating the degree of matching of the target detection frames of two adjacent frames of images to be detected to determine whether they are the same person target, thereby obtaining a person tracking result.

[0044] Step S14, determining the corresponding sleeping behavior detection result of each person according to the sleeping posture detection result and the personnel tracking result.

[0045] In this embodiment, reference Figure 2, according to the sleeping posture detection results and the personnel tracking results, the corresponding sleeping behavior detection results of each person are determined, including: according to the sleeping posture detection results and the personnel tracking results, the sleeping posture recognition results of each person target in each frame of the image to be detected are obtained; according to the sleeping posture recognition results of each person target in all frames of the image to be detected within the target time, the number of results of the sleeping posture recognition results of each person is determined to be the sleeping category; compare the number of results with the preset sleeping posture threshold, and determine that the person status corresponding to the target time is the sleeping posture state based on the number of results being greater than the preset sleeping posture threshold; otherwise, it is the non-sleeping posture state; according to the person status corresponding to the target time and the historical status score obtained in advance, the sleeping behavior detection result of the corresponding person is obtained; wherein the historical status score is obtained based on the cumulative score of the person status determined at the historical target time prior to the target time.

[0046] Furthermore, the historical status score includes a sleeping on duty behavior score and an interruption timing score. According to the personnel status corresponding to the target time and the historical status score obtained in advance, the sleeping on duty behavior detection result of the corresponding personnel is obtained, including: if the personnel status corresponding to the target time is the sleeping on duty state, then the sleeping on duty behavior of the corresponding personnel is timed, and the sleeping on duty behavior score of the corresponding personnel is increased by one point and the interruption timing score is reduced by one point; otherwise, the sleeping on duty behavior of the corresponding personnel is interrupted, and the interruption timing score of the corresponding personnel is increased by one point and the sleeping on duty behavior score is reduced by one point, so as to obtain the sleeping on duty behavior detection result of the corresponding personnel, thereby monitoring the sleeping on duty behavior of the personnel according to the sleeping on duty behavior detection result to reduce the occurrence rate of safety accidents.

[0047] In an optional embodiment, after obtaining the corresponding person's sleeping on duty behavior detection result, it also includes: determining whether the current interruption timing score exceeds the interruption threshold, then resetting the state, resetting the current sleeping on duty behavior score and the current interruption timing score to zero, so as to facilitate automatic resetting of the timing when the user's posture changes, avoiding false alarms caused by short-term posture adjustments.

[0048] In an optional embodiment, after obtaining the detection result of the corresponding personnel's sleeping on duty behavior, it includes: based on the detection result of the corresponding personnel's sleeping on duty behavior, combined with a preset violation handling strategy, the corresponding personnel is handled for violation, so as to reduce the occurrence rate of safety accidents and ensure the safety of on-site personnel and equipment; wherein, the preset violation handling strategy is used to limit the violation handling measures corresponding to the sleeping on duty behavior detection result.

[0049] It should be added that the preset violation handling strategy can be set according to the actual violation handling needs. For example, if the target time is 1 minute and the person is sleeping on duty within 5 minutes, the corresponding preset violation handling strategy can be to send an alarm reminder based on the score of sleeping on duty being less than 5, and to make background records; if the person continues to sleep on duty and the duration exceeds 10 minutes, the corresponding preset violation handling strategy can be to send an alarm reminder and push information to the security control personnel based on the score of sleeping on duty being greater than 10, and to make background records. This is only an example, and it can be configured according to the actual monitoring scenario involved and the degree of danger to personnel safety caused by sleeping on duty, and no further limitation is made here. In addition, the alarm reminder can be sent based on an audible and visual alarm.

[0050] To summarize, the embodiments of the present invention perform sleeping posture detection on the images to be detected within the target time through a sleeping posture detection model to avoid misjudgment caused by single frame image detection, thereby more accurately judging whether a person is in a sleeping state and improving the accuracy of detection, and further combining the sleeping posture detection results to track the person and realize continuous monitoring of the person's status, thereby determining the sleeping behavior detection results corresponding to each person based on the sleeping posture detection results and the person tracking results, comprehensively evaluating whether the person has sleeping behavior and reducing the possibility of misjudgment.

[0051] The following is a description of the post sleeping behavior detection device provided by the present invention. The post sleeping behavior detection device described below and the post sleeping behavior detection method described above can be referenced to each other.

[0052] Figure 3 The structure diagram of a device for detecting post sleeping behavior is shown, and the device comprises: An image acquisition module 31 is used to acquire the image to be detected within a target time; The sleeping posture detection module 32 inputs the image to be detected within the target time into the sleeping posture detection model, and obtains the sleeping posture detection result output by the sleeping posture detection model; wherein the sleeping posture detection model is trained based on the training image data set and the detection label corresponding to the training image data set, and the sleeping posture detection model is used to perform feature fusion based on the image features extracted from the input image to be detected within the target time, and perform target detection on the fused features to obtain the sleeping posture detection result; The personnel tracking module 33 tracks the personnel according to the sleeping posture detection result to obtain the personnel tracking result; The behavior monitoring module 34 determines the corresponding sleeping behavior detection result of each person according to the sleeping posture detection result and the personnel tracking result.

[0053] In this embodiment, the image acquisition module 31 includes: a first image acquisition unit, which shoots the target monitoring area based on a preset shooting interval to obtain a corresponding image to be detected and a corresponding shooting time; based on the image to be detected and the corresponding shooting time; a second image acquisition unit, which determines the shooting duration based on the shooting time and the preset shooting interval, and based on the shooting duration being consistent with the target time, uses the corresponding image to be detected as the image to be detected within the target time.

[0054] Furthermore, when photographing the target monitoring area based on a preset shooting interval, the shooting angle can be adjusted based on a preset adjustment frequency to obtain images to be detected at different viewing angles.

[0055] In addition, when photographing the target monitoring area, it also includes: photographing the target detection area using at least two camera devices with different shooting angles to obtain corresponding images to be detected.

[0056] The sleeping posture detection module 32 includes: a feature extraction unit, which inputs the image to be detected within the target time into the feature extraction layer for feature extraction, and obtains the image features corresponding to each frame of the image output by the feature extraction layer; a feature fusion unit, which inputs the image features corresponding to each frame of the image into the feature fusion layer, so as to perform feature fusion on the corresponding image features for each frame of the image to be detected, and obtain the fusion features corresponding to each frame of the image to be detected output by the feature fusion layer; a target detection unit, which inputs the fusion features corresponding to each frame of the image to be detected into the target detection layer for sleeping posture detection, and obtains the sleeping posture detection result output by the target detection layer.

[0057] In an optional embodiment, the device further includes: a training data acquisition module, which acquires a training image data set before inputting the images to be detected within the target time into the sleeping position detection model, and the training image data set includes multiple training image frame sequences; a label acquisition module, which acquires the detection label of each frame of the training image corresponding to each image frame sequence, and the detection label includes a sleeping position category label, a person position label, and a target situation label of whether the person position determined according to the person position label contains the corresponding person; a forward propagation module, which selects a preset number of image frame sequences from the training image data set for each iterative training, and inputs the selected image frame sequences into the model to be trained respectively, to obtain the sleeping position prediction results of the corresponding image frame sequences output by the model to be trained, and the sleeping position prediction results include the sleeping position detection results, the person position label, and the target situation label of whether the corresponding person is contained in the person position determined according to the person position label; The detection box and the confidence of the corresponding person included in the person detection box; the loss calculation module constructs a category loss function according to the sleeping posture detection result and the corresponding sleeping posture category label, constructs a position loss function according to the person detection box and the corresponding person position label, and constructs a confidence loss function according to the confidence that the corresponding person is included in the person detection box and the corresponding target situation label; the total loss function is obtained according to the category loss function, the position loss function and the confidence loss function; the back propagation module determines the gradient information of the total loss function based on the back propagation algorithm; according to the gradient information of the total loss function, the preset optimization algorithm is used to update the network parameters of the model to be trained, and the next iterative training is performed until the number of iterations meets the preset training number or the corresponding total loss function converges, so as to obtain a trained sleeping posture detection model.

[0058] Specifically, the training data acquisition module is used to: photograph the monitoring area using multiple photographing devices based on a preset time interval, and obtain a target number of image frame sequences based on the shooting duration being consistent with the target time; and construct a training image data set based on the target number of image frame sequences. In addition, the training data acquisition module is also used to: when photographing the monitoring area, it can also be acquired based on different attitude angles, obstructions, lighting, weather and other factors to improve the robustness of subsequent model training.

[0059] Furthermore, the device also includes: a preprocessing module, which preprocesses the captured images after the monitoring area is photographed using multiple photographing devices, and the preprocessing includes at least one of denoising and color correction.

[0060] In order to avoid redundancy between consecutive frame training images, the device also includes: a grouping module, after using multiple shooting devices to shoot the monitoring area, based on a preset time length, divides the training image frame sequence shot based on a preset time interval to obtain different groups; a screening module, identifies the training images in each group to eliminate images that do not contain human targets.

[0061] In addition, the label acquisition module is used to: after obtaining the training image data set, mark the person position of each training image in the training image data set, the sleeping posture of the corresponding person, and whether the corresponding person is contained in the corresponding person position, to obtain the sleeping posture category label, the person position label and the target situation label.

[0062] In addition, the device also includes: a data enhancement module, which, after obtaining the training image data set, uses a data enhancement strategy to perform data enhancement on the training images in the training image data set.

[0063] In an optional embodiment, the device further includes: a label assignment module, which assigns a detection label to the sleeping posture prediction result based on a preset label assignment strategy before constructing a category loss function according to the sleeping posture detection result and the corresponding sleeping posture category label, constructing a position loss function according to the person detection frame and the corresponding person position label, and constructing a confidence loss function according to the confidence that the corresponding person is included in the person detection frame and the corresponding target situation label.

[0064] Specifically, the sleeping posture prediction result includes the sleeping posture prediction result of each training image in the training image frame sequence; the label assignment module includes: a grid division unit, according to each training image in the training image frame sequence corresponding to the sleeping posture prediction result, grid division of each training image to obtain a corresponding grid unit; an estimation unit, based on a preset person size range, preset position and a preset position change range of a person between adjacent frames, estimates the sleeping posture category and the number of people contained in each grid unit of each training image frame; an area selection unit, according to the estimated target category and number contained in each grid unit of each training image frame, determines the sleeping posture category and the number of people contained in each grid unit of each training image frame. A candidate target area of ​​the image, wherein the candidate target area is composed of grid units containing the target determined based on the sleeping posture category and the number of people contained in each grid unit; a coarse matching unit, for each candidate target area, determines the intersection-over-union ratio of the candidate target area and the sleeping posture category label, and based on a first preset threshold, screens out candidate target areas that are not greater than the first preset threshold to obtain a coarse matching area; a label assignment unit, based on the target semantic features and context information obtained by the auxiliary branch contained in the YOLOv7 network structure, combined with preset data feature matching conditions, evaluates the matching degree between the coarse matching area and the sleeping posture category label, and obtains a label assignment result.

[0065] The personnel tracking module 33 includes: an intersection-and-union ratio acquisition unit, which obtains the intersection-and-union ratio information based on the target detection frames of two adjacent frames of images to be detected; a personnel matching unit, which evaluates the matching degree of the target detection frames of two adjacent frames of images to be detected based on the intersection-and-union ratio information, determines whether they are the same person target, and obtains the personnel tracking result.

[0066] The behavior monitoring module 34 includes: a person sleeping posture determination unit, which obtains the sleeping posture recognition result of each person target in each frame of the image to be detected according to the sleeping posture detection result and the person tracking result; a quantity determination unit, which determines the number of results of the sleeping posture recognition result of each person target in all frames of the image to be detected within the target time, according to the sleeping posture recognition result of each person target in all frames of the image to be detected; a sleeping guard state judgment unit, which compares the number of results with a preset sleeping guard threshold, and determines that the person state corresponding to the target time is the sleeping guard state based on the number of results being greater than the preset sleeping guard threshold; otherwise, it is the non-sleeping guard state; a sleeping guard result determination unit, which obtains the sleeping guard behavior detection result of the corresponding person according to the person state corresponding to the target time and the historical state score obtained previously; wherein the historical state score is obtained based on the cumulative score of the person state determined at the historical target time prior to the target time.

[0067] Furthermore, the historical status score includes the post sleeping behavior score and the post interruption timing score. The post sleeping result determination unit is used to: if the personnel status corresponding to the target time is the post sleeping status, then the post sleeping behavior of the corresponding personnel is timed, and the post sleeping behavior score of the corresponding personnel is increased by one point and the interruption timing score is reduced by one point; otherwise, the post sleeping behavior of the corresponding personnel is interrupted, and the interruption timing score of the corresponding personnel is increased by one point and the post sleeping behavior score is reduced by one point, so as to obtain the post sleeping behavior detection result of the corresponding personnel.

[0068] In an optional embodiment, the behavior monitoring module 34 also includes: a reset unit, which, after obtaining the corresponding person's sleeping behavior detection result, determines whether the current interruption timing score exceeds the interruption threshold, and then resets the state to reset the current sleeping behavior score and the current interruption timing score to zero, so as to automatically reset the timing when the user's posture changes, avoiding false alarms caused by short-term posture adjustments.

[0069] In an optional embodiment, the device also includes: a violation handling module, after obtaining the detection result of the corresponding personnel's sleeping on the job behavior, performs violation handling on the corresponding personnel according to the detection result of the corresponding personnel's sleeping on the job behavior and in combination with a preset violation handling strategy, so as to reduce the occurrence rate of safety accidents and ensure the safety of on-site personnel and equipment; wherein the preset violation handling strategy is used to limit the violation handling measures corresponding to the sleeping on the job behavior detection result.

[0070] To summarize, the embodiments of the present invention use a sleeping posture detection model through a sleeping posture detection module to perform sleeping posture detection on the images to be detected within the target time acquired by the image acquisition module, so as to avoid misjudgment caused by single frame image detection, thereby more accurately judging whether a person is in a sleeping state and improving the accuracy of detection, and further tracking the person through a personnel tracking module in combination with the sleeping posture detection result to achieve continuous monitoring of the person's status, thereby facilitating the behavior monitoring module to determine the sleeping behavior detection result corresponding to each person according to the sleeping posture detection result and the personnel tracking result, comprehensively evaluating whether the person has sleeping behavior and reducing the possibility of misjudgment.

[0071] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the method for detecting the behavior of sleeping on the post, which includes: obtaining the image to be detected within the target time; inputting the image to be detected within the target time into the sleeping position detection model, and obtaining the sleeping position detection result output by the sleeping position detection model; wherein the sleeping position detection model is trained based on the training image data set and the detection label corresponding to the training image data set, and the sleeping position detection model is used to perform feature fusion based on the image features extracted from the input image to be detected within the target time, and perform target detection on the fused features to obtain the sleeping position detection result; according to the sleeping position detection result, track the person to obtain the person tracking result; according to the sleeping position detection result and the person tracking result, determine the sleeping position behavior detection result corresponding to each person.

[0072] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0073] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the sleeping post behavior detection method provided by the above-mentioned methods, which includes: obtaining images to be detected within a target time; inputting the images to be detected within the target time into a sleeping position detection model, and obtaining a sleeping position detection result output by the sleeping position detection model; wherein the sleeping position detection model is trained based on a training image data set and detection labels corresponding to the training image data set, and the sleeping position detection model is used to perform feature fusion based on image features extracted from the input images to be detected within the target time, and perform target detection on the fused features to obtain sleeping position detection results; according to the sleeping position detection results, track personnel to obtain personnel tracking results; and determine the sleeping post behavior detection results corresponding to each person based on the sleeping position detection results and the personnel tracking results.

[0074] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the sleeping post behavior detection method provided by the above-mentioned methods, the method comprising: obtaining images to be detected within a target time; inputting the images to be detected within the target time into a sleeping position detection model, and obtaining a sleeping position detection result output by the sleeping position detection model; wherein the sleeping position detection model is trained based on a training image data set and detection labels corresponding to the training image data set, and the sleeping position detection model is used to perform feature fusion based on image features extracted from the input images to be detected within the target time, and perform target detection on the fused features to obtain sleeping position detection results; based on the sleeping position detection results, tracking personnel to obtain personnel tracking results; and determining the sleeping post behavior detection results corresponding to each person based on the sleeping position detection results and the personnel tracking results.

[0075] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting post sleeping behavior, characterized in that: include: Acquire the image to be detected within the target time; Input the image to be detected within the target time into the sleeping position detection model, and obtain the sleeping position detection result output by the sleeping position detection model; wherein the sleeping position detection model is trained based on a training image data set and a detection label corresponding to the training image data set, and the sleeping position detection model is used to perform feature fusion based on image features extracted from the input image to be detected within the target time, and perform target detection on the fused features to obtain the sleeping position detection result; Tracking a person according to the sleeping posture detection result to obtain a person tracking result; According to the sleeping posture detection result and the personnel tracking result, the sleeping-on-the-job behavior detection result corresponding to each person is determined.

2. The method for detecting post sleeping behavior according to claim 1, characterized in that: The sleeping posture detection model comprises: The feature extraction layer extracts features from the image to be detected within the input target time to obtain the image features corresponding to each frame of the image; The feature fusion layer performs feature fusion on the corresponding image features for each frame of the image to be detected to obtain the fusion features corresponding to each frame of the image to be detected; The target detection layer performs sleeping posture detection on the fused features corresponding to the image to be detected in each frame, and uses non-maximum suppression to obtain the corresponding sleeping posture detection result.

3. The method for detecting post sleeping behavior according to claim 1, characterized in that: Before inputting the image to be detected within the target time into the sleeping posture detection model, the method includes: Acquire a training image data set, wherein the training image data set includes a plurality of training image frame sequences; Obtaining a detection label for each training image frame corresponding to each of the image frame sequences, the detection label comprising a sleeping posture category label, a person position label, and a target situation label of whether the person position determined according to the person position label contains a corresponding person; For each iterative training, a preset number of image frame sequences are selected from the training image data set, and the selected image frame sequences are respectively input into the to-be-trained model to obtain sleeping posture prediction results of the corresponding image frame sequences output by the to-be-trained model, wherein the sleeping posture prediction results include sleeping posture detection results, person detection frames, and confidence that the corresponding person is included in the person detection frames; Constructing a category loss function according to the sleeping posture detection result and the corresponding sleeping posture category label, constructing a position loss function according to the person detection frame and the corresponding person position label, and constructing a confidence loss function according to the confidence that the corresponding person is included in the person detection frame and the corresponding target situation label; Obtaining a total loss function according to the category loss function, the position loss function and the confidence loss function, and determining the gradient information of the total loss function based on a back propagation algorithm; According to the gradient information of the total loss function, the preset optimization algorithm is used to update the network parameters of the model to be trained, and the next iterative training is performed until the number of iterations meets the preset number of training times or the corresponding total loss function converges, thereby obtaining a trained sleeping posture detection model.

4. The method for detecting post sleeping behavior according to claim 1, characterized in that: The sleeping posture detection result includes the target detection frame of each frame of the image to be detected within the target time. According to the sleeping posture detection result, the person is tracked to obtain the person tracking result, including: According to the target detection frames of two adjacent frames of the image to be detected, the intersection-and-union ratio information is obtained; According to the intersection-over-union ratio information, the matching degree of the target detection frames of the two adjacent frames of the to-be-detected images is evaluated to determine whether they are the same person target, and obtain the person tracking result.

5. The method for detecting post sleeping behavior according to claim 1, characterized in that: The sleeping posture detection result includes the sleeping posture recognition result of each frame of the image to be detected within the target time. According to the sleeping posture detection result and the personnel tracking result, the sleeping behavior detection result corresponding to each person is determined, including: Obtaining a sleeping posture recognition result of each person target in each frame of the image to be detected according to the sleeping posture detection result and the person tracking result; According to the sleeping posture recognition results of each person target in all frames of the to-be-detected image within the target time, determine the number of results of the sleeping posture recognition results of each person that are in the sleeping category; Compare the number of results with a preset threshold value for sleeping on duty, and determine that the state of the personnel corresponding to the target time is a sleeping state based on the number of results being greater than the preset threshold value for sleeping on duty; otherwise, it is a non-sleeping state; According to the personnel status corresponding to the target time and the historical status score obtained previously, the corresponding personnel sleeping on duty behavior detection result is obtained; wherein, the historical status score is obtained based on the cumulative score of the personnel status determined at the historical target time prior to the target time.

6. The method for detecting post sleeping behavior according to claim 5, characterized in that: The historical status score includes a post sleeping behavior score and an interruption timing score. According to the personnel status corresponding to the target time and the previously obtained historical status score, the corresponding personnel's post sleeping behavior detection result is obtained, including: If the personnel status corresponding to the target time is the sleeping status, the sleeping behavior of the corresponding personnel is timed, and the sleeping behavior score of the corresponding personnel is increased by one point and the interruption timing score is reduced by one point; otherwise, the sleeping behavior of the corresponding personnel is interrupted and timed, and the interruption timing score of the corresponding personnel is increased by one point and the sleeping behavior score is reduced by one point, so as to obtain the sleeping behavior detection result of the corresponding personnel.

7. The method for detecting post sleeping behavior according to claim 1, characterized in that: After obtaining the corresponding personnel's sleeping behavior detection results, including: According to the detection results of the corresponding personnel's sleeping on the job behavior and in combination with the preset violation handling strategy, the corresponding personnel are handled for violation; wherein, the preset violation handling strategy is used to limit the violation handling measures corresponding to the detection results of the sleeping on the job behavior.

8. A device for detecting post sleeping behavior, characterized in that: include: An image acquisition module acquires the image to be detected within a target time; A sleeping posture detection module, inputting the image to be detected within the target time into the sleeping posture detection model, and obtaining the sleeping posture detection result output by the sleeping posture detection model; wherein the sleeping posture detection model is trained based on a training image data set and a detection label corresponding to the training image data set, and the sleeping posture detection model is used to perform feature fusion based on image features extracted from the input image to be detected within the target time, and perform target detection on the fused features to obtain the sleeping posture detection result; A personnel tracking module, which tracks the personnel according to the sleeping posture detection result to obtain the personnel tracking result; The behavior monitoring module determines the corresponding sleeping behavior detection result of each person according to the sleeping posture detection result and the personnel tracking result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for detecting post sleeping behavior as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for detecting post sleeping behavior as described in any one of claims 1 to 7 is implemented.

Images