Sleep behavior detection method and device and electronic equipment

Abstract

The invention provides a sleep behavior detection method and device and electronic equipment, and the method comprises the steps: carrying out the detection of a current video image for each frame of video image in a to-be-detected video, and obtaining a detection result; if it is determined that at least one first detection frame exists according to the detection result, obtaining a camera identifier corresponding to the current video image; if it is determined that historical time information and at least one historical detection frame of a previous frame of video image corresponding to the camera identifier exist according to the camera identifier, detecting whether a sleep behavior exists through a DHash algorithm based on current time information and historical time information of a current video image, each first detection frame and each historical detection frame; according to the mode, the current time information of the current video image, each first detection frame, and the historical time information and each historical detection frame of the previous frame of video image can be combined, and the DHash algorithm is used to detect whether the sleep behavior exists, so that the detection accuracy of the sleep behavior can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method, device and electronic equipment for detecting post sleeping behavior. Background Art

[0002] Enterprises usually equip various production environments with video systems to monitor the safety production conditions in the video. Usually, enterprises will have 24-hour monitoring personnel in the central control room to monitor and analyze the video content. However, the sleeping behavior of monitoring personnel is prone to negligence and may cause major safety hazards, so it is necessary to analyze the behavior of monitoring personnel. In related technologies, a single-frame image can be used to detect sleeping behavior through a combination of detection + classification. This method is not rigorous enough and has a high false alarm rate. Personnel tracking can also be achieved through the YOLOv5+deepsort algorithm, and facial information can be obtained through RetinaFace. By setting certain thresholds, logical judgments are made on human body information and facial information to identify sleeping behavior. In this method, the deepsort algorithm may easily swap IDs due to occlusion or human interaction in the actual environment, resulting in confused results. The use of facial information by setting a threshold often requires the camera to be facing the person, while the position of the camera is not fixed in the actual generation process, and in many cases it is far away from the person, resulting in low face resolution and difficulty in recognition, and the face cannot be detected for people who sleep on their stomachs. In summary, related technologies generally have the problem of low detection accuracy for sleeping behavior. Summary of the invention

[0003] The purpose of the present invention is to provide a method, device and electronic equipment for detecting post sleeping behavior, so as to improve the detection accuracy of post sleeping behavior.

[0004] The present invention provides a method for detecting post sleeping behavior, the method comprising: obtaining a video to be detected; for each frame of video image in the video to be detected, detecting the current video image to obtain a detection result corresponding to the current video image; wherein the detection result includes: a category label and a behavior label corresponding to each target in the current video image; if it is determined according to the detection result that there is at least one first detection frame, obtaining a camera identifier corresponding to the current video image; wherein the category label corresponding to the target in each first detection frame is a human label, and the corresponding behavior label is a label associated with sleeping behavior;

[0005] If it is determined based on the camera identification that there is historical time information and at least one historical detection frame of the previous frame of the video image corresponding to the camera identification, based on the current time information of the current video image, the historical time information, each first detection frame and each historical detection frame, the DHash algorithm is used to detect whether there is sleeping on the post behavior; wherein the category label corresponding to the target in each historical detection frame is a human label, and the corresponding behavior label is a label associated with sleeping behavior.

[0006] Furthermore, if it is determined based on the detection results that there is at least one first detection frame, the step of obtaining the camera identification corresponding to the current video image includes: judging based on the detection results whether there is a target behavior label associated with sleeping behavior in the current video image; if there is at least one target behavior label, cropping the image area where the target corresponding to each target behavior label is located to obtain a second detection frame of the target corresponding to each target behavior label; judging based on the category label corresponding to the target in each second detection frame whether there is at least one first detection frame whose category label is a human label; if there is at least one first detection frame, obtaining the camera identification corresponding to the current video image.

[0007] Furthermore, the method also includes: if there is no at least one first detection frame, determining that the target in each second detection frame is a static object in a sleeping posture.

[0008] Furthermore, if it is determined according to the camera identifier that there is historical time information and at least one historical detection frame of the previous frame of the video image corresponding to the camera identifier, the step of detecting whether there is a post sleeping behavior by using the DHash algorithm based on the current time information, the historical time information, each first detection frame and each historical detection frame of the current video image includes:

[0009] According to the camera identification, determine whether there is historical time information and at least one historical detection frame of the previous frame of video image corresponding to the camera identification; if there is historical time information and at least one historical detection frame, calculate the first time interval according to the historical time information and the current time information of the current video image; if the first time interval is not greater than the preset time interval threshold, traverse each first detection frame and calculate the intersection and union ratio of each first detection frame and each historical detection frame; based on the intersection and union ratio, detect whether there is post sleeping behavior through the DHash algorithm.

[0010] Furthermore, the method also includes: if there is no historical time information and at least one historical detection frame, initializing the time of the camera corresponding to the camera identifier to the current time information.

[0011] Furthermore, the method also includes: if the first time interval is greater than a preset time interval threshold, determining that there is no post sleeping behavior.

[0012] Furthermore, based on the intersection-and-union ratio, the step of detecting whether there is a post-sleeping behavior through the DHash algorithm includes: if there is at least one detection frame pair whose intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold, determining whether the first time interval is greater than a preset alarm time threshold; if the first time interval is greater than the alarm time threshold, calculating, for each detection frame pair, the similarity between the target in the first detection frame of the target and the target in the historical detection frame of the target in the detection frame pair through the DHash algorithm; and determining whether there is a post-sleeping behavior based on the result of the similarity.

[0013] Furthermore, based on the similarity result, the step of determining whether there is any sleeping on the post behavior includes: if the similarity is less than a preset similarity threshold, confirming that the target in the first target detection frame and the target in the target history detection frame are the same target, determining that there is any sleeping on the post behavior, and generating a sleeping on the post behavior alarm message; if the similarity is not less than the similarity threshold, determining that there is no sleeping on the post behavior.

[0014] The present invention provides a device for detecting post sleeping behavior, the device comprising: a first acquisition module, used to acquire a video to be detected; a first detection module, used to detect the current video image for each frame of video image in the video to be detected, and obtain a detection result corresponding to the current video image; wherein the detection result includes: a category label and a behavior label corresponding to each target in the current video image; a second acquisition module, used to obtain a camera identifier corresponding to the current video image if it is determined that there is at least one first detection frame according to the detection result; wherein the category label corresponding to the target in each first detection frame is a human label, and the corresponding behavior label is a label associated with sleeping behavior;

[0015] The second detection module is used to determine, based on the camera identification, the existence of historical time information and at least one historical detection frame of the previous frame of the video image corresponding to the camera identification, and detect whether there is sleeping on the post through the DHash algorithm based on the current time information of the current video image, the historical time information, each first detection frame and each historical detection frame; wherein the category label corresponding to the target in each historical detection frame is a human label, and the corresponding behavior label is a label associated with sleeping behavior.

[0016] An electronic device provided by the present invention includes a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement any of the above-mentioned methods for detecting post sleeping behavior.

[0017] The method, device and electronic device for detecting the behavior of sleeping on the post provided by the present invention obtain a video to be detected; for each frame of video image in the video to be detected, the current video image is detected to obtain the detection result corresponding to the current video image; wherein the detection result includes: the category label and behavior label corresponding to each target in the current video image; if it is determined according to the detection result that there is at least one first detection frame, the camera identification corresponding to the current video image is obtained; wherein the category label corresponding to the target in each first detection frame is a human label, and the corresponding behavior label is a label associated with the sleeping behavior; if it is determined according to the camera identification that there is the historical time information of the previous frame of video image corresponding to the camera identification and at least one historical detection frame, based on the current time information, historical time information, each first detection frame and each historical detection frame of the current video image, the DHash algorithm is used to detect whether there is the behavior of sleeping on the post; wherein the category label corresponding to the target in each historical detection frame is a human label, and the corresponding behavior label is a label associated with the sleeping behavior. This method can combine the current time information of the current video image, each first detection frame, and the historical time information and each historical detection frame of the previous frame of video image, and use the DHash algorithm to detect whether there is the behavior of sleeping on the post, which can improve the detection accuracy of the behavior of sleeping on the post. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of a method for detecting post sleeping behavior provided by an embodiment of the present invention;

[0020] Figure 2 A flowchart of generating a hash value using a DHash algorithm provided in an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the structure of a post sleeping behavior detection system provided by an embodiment of the present invention;

[0022] Figure 4 A flow chart of another method for detecting post sleeping behavior provided by an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of the structure of a device for detecting post sleeping behavior provided by an embodiment of the present invention;

[0024] Figure 6A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] In modern industry, production safety has always been a top priority for management of various enterprises. Lack of sleep or fatigue of employees is one of the causes of accidents. In order to prevent the decline of employee work efficiency and to reduce the safety hazards of enterprises, the technology of detecting sleeping on the job has emerged. Enterprises usually equip video systems in various production environments and equip the central control room with 24-hour monitoring personnel to monitor and analyze the video content. However, the sleeping behavior of monitoring personnel is prone to negligence and may cause major safety hazards. Therefore, the behavior analysis of monitoring personnel in the central control room is of great significance. The related technology discloses a method of detecting sleeping on the job based on deep learning. In terms of design concept, this method solves the problem of low accuracy of sleeping on the job behavior by combining detection + classification. The process is relatively cumbersome, and the data production is too troublesome. The video images used for collection and detection must also be cropped out of the video and a classification data set must be produced; and the method only considers the detection results of a single frame. This method is not rigorous enough and is prone to many false alarms.

[0027] Related technologies also disclose another method for identifying sleeping behavior of on-duty personnel in the field of prosecution. This method uses the YOLOv5+deepsort algorithm to achieve personnel tracking, obtains facial information through RetinaFace, and sets a certain threshold to perform logical judgment on human body information and facial information to identify sleeping behavior. This method first has certain disadvantages in speed performance. It needs to be on a high-performance GPU (Graphics Processing Unit) to meet the startup requirements of the algorithm, but it is still difficult to meet the requirements of high frame rate. At the same time, this method does not say how to solve the occlusion situation in the actual environment. The deepsort algorithm can easily exchange IDs due to occlusion or personnel interaction in the actual environment, resulting in confusion of results. Using facial information, this method of setting a threshold often requires the position of the camera to be facing the person, but in the actual generation process, the position of the camera is not fixed, and in many cases, the camera is far away from the person, resulting in low facial resolution and difficulty in recognition. In addition, the face cannot be detected for people sleeping on their stomachs, so the functionality of the algorithm is questionable.

[0028] In summary, the traditional monitoring methods in the relevant technologies require manual auditing, which is time-consuming, labor-intensive, inefficient and costly. In addition, during image recognition detection, it is also affected by factors such as facial occlusion, posture diversity, and angle changes, resulting in low recognition accuracy. Based on this, the embodiments of the present invention provide a method, device and electronic device for detecting sleeping on the job, which can be applied to applications that require detection of sleeping on the job in images.

[0029] To facilitate understanding of this embodiment, firstly, a method for detecting post sleeping behavior disclosed in an embodiment of the present invention is introduced. Figure 1 As shown, the method comprises the following steps:

[0030] Step S102, obtaining a video to be detected;

[0031] The above-mentioned video to be detected may be a video collected by a camera or other device, and the video to be detected may be a video of the enterprise's central control room, duty room, etc., which usually includes monitoring personnel in the central control room and duty room; the monitoring personnel may monitor the video content of various production environments of the enterprise through monitoring equipment installed in the central control room and duty room; in actual implementation, when it is necessary to detect the sleeping behavior of monitoring personnel in the central control room, duty room, etc., it is usually necessary to obtain the corresponding video to be detected first.

[0032] Step S104, for each frame of the video image in the video to be detected, the current video image is detected to obtain a detection result corresponding to the current video image; wherein the detection result includes: a category label and a behavior label corresponding to each target in the current video image;

[0033] The above-mentioned targets can be understood as targets such as people, tables, and seats contained in the current video image; the above-mentioned category labels can be used to indicate the category to which the target belongs, for example, the category label can be a human label, a table label, a seat label, etc.; the above-mentioned behavior label can be used to indicate the behavior of the target, for example, it can be sleeping behavior, standing behavior, etc.; in actual implementation, after obtaining the video to be detected, the current video image can be detected for each frame of video image in the video to be detected to identify the category, behavior, etc. of each target in the current video image, and obtain the category label and behavior label corresponding to each target; the category labels corresponding to different targets may be the same or different, and the behavior labels corresponding to different targets may be the same or different.

[0034] Step S106: if it is determined according to the detection result that there is at least one first detection frame, a camera identifier corresponding to the current video image is obtained; wherein the category label corresponding to the target in each first detection frame is a human label, and the corresponding behavior label is a label associated with sleeping behavior;

[0035] The above-mentioned camera identification can be the ID (Identity document) of the camera, etc.; corresponding camera identifications can be set for different cameras to avoid confusion between different cameras. In actual implementation, based on the above-mentioned detection results of the current video image, it can be judged whether there is a target with a category label of human and a behavior label associated with sleeping behavior in the current video image. If there is a target that meets the requirements, it can be determined that there is a first detection frame corresponding to the target, wherein there may be one or more targets that meet the requirements. If there is one target that meets the requirements, there is also one first detection frame corresponding to the target. If there are multiple targets that meet the requirements, there are also multiple first detection frames corresponding to each target. In actual implementation, after determining that there are one or more first detection frames according to the detection results, the camera identification of the camera that captures the current video image can be obtained.

[0036] Step S108, if it is determined based on the camera identification that there is historical time information and at least one historical detection frame of the previous frame of the video image corresponding to the camera identification, based on the current time information of the current video image, the historical time information, each first detection frame and each historical detection frame, the DHash algorithm is used to detect whether there is a sleeping behavior; wherein the category label corresponding to the target in each historical detection frame is a human label, and the corresponding behavior label is a label associated with sleeping behavior.

[0037] The above-mentioned previous frame of video image can be understood as a frame of video image saved before obtaining the current video image. The category label in the previous frame of video image is a human label, and the behavior label is a target associated with sleeping behavior. It can be understood that the previous frame of video image may be a previous frame of video image that is an adjacent frame to the current video image in the video to be detected, or it may be a frame of video image that is multiple frames apart from the current video image, which is usually related to whether the person in the video to be detected has sleeping behavior; the above-mentioned historical time information can be understood as the image acquisition time corresponding to the previous frame of video image; the method for determining the above-mentioned historical detection frame can refer to the method for determining the above-mentioned first detection frame. Formula; the above current time information can be understood as the image acquisition time corresponding to the current video image; the DHash algorithm is a differential hash algorithm. When performing hash conversion on an image, the final hash sequence is obtained by comparing the sizes of two pixels on the left and right; in actual implementation, after obtaining the above camera identification, it is possible to determine whether there is pre-stored historical time information and at least one historical detection frame of the previous frame of video image acquired by the same camera based on the camera identification. If so, the DHash algorithm can be used to detect whether there is any sleeping behavior based on the current time information, historical time information, each first detection frame and each historical detection frame of the current video image.

[0038] The above-mentioned method for detecting the behavior of sleeping on the post obtains a video to be detected; for each frame of video image in the video to be detected, the current video image is detected to obtain the detection result corresponding to the current video image; wherein the detection result includes: the category label and behavior label corresponding to each target in the current video image; if it is determined according to the detection result that there is at least one first detection frame, the camera identification corresponding to the current video image is obtained; wherein the category label corresponding to the target in each first detection frame is a human label, and the corresponding behavior label is a label associated with the sleeping behavior; if it is determined according to the camera identification that there is the historical time information of the previous frame of video image corresponding to the camera identification and at least one historical detection frame, based on the current time information, historical time information, each first detection frame and each historical detection frame of the current video image, the DHash algorithm is used to detect whether there is the behavior of sleeping on the post; wherein the category label corresponding to the target in each historical detection frame is a human label, and the corresponding behavior label is a label associated with the sleeping behavior. This method can combine the current time information of the current video image, each first detection frame, and the historical time information and each historical detection frame of the previous frame of video image, and use the DHash algorithm to detect whether there is the behavior of sleeping on the post, which can improve the detection accuracy of the behavior of sleeping on the post.

[0039] The embodiment of the present invention further provides another method for detecting post sleeping behavior, which is implemented on the basis of the method in the above embodiment, and comprises the following steps:

[0040] Step 1: Obtain the video to be detected;

[0041] Step 2: for each frame of the video to be detected, the current video image is detected to obtain the detection result corresponding to the current video image; wherein the detection result includes: the category label and behavior label corresponding to each target in the current video image;

[0042] Step 3: judging whether there is a target behavior label associated with sleeping behavior in the current video image according to the detection result;

[0043] In actual implementation, since the above detection results contain the category label, behavior label, etc. corresponding to each target in the current video image, screening and filtering can be performed based on the detection results to determine whether there is a target behavior label associated with sleeping behavior.

[0044] Step 4: if there is at least one target behavior label, crop the image area where the target corresponding to each target behavior label is located to obtain a second detection frame of the target corresponding to each target behavior label;

[0045] If it is determined through screening and filtering that there are one or more target behavior labels associated with sleeping behavior, the image area where the target corresponding to the target behavior label is located can be cropped. The image area can be a rectangular area containing the target, or it can be an area cropped according to the outline of the target. The specific selection can be made according to actual needs and is not limited here, so as to obtain a second detection frame of the target corresponding to each target behavior label.

[0046] Step 5: judging whether there is at least one first detection frame whose category label is a human label according to the category label corresponding to the target in each second detection frame;

[0047] After cropping each second detection frame, an asynchronous request processing method can be used to determine whether each category label is a human label based on the category label corresponding to the target in each second detection frame, so as to further determine whether there is at least one first detection frame with a category label of a human label in all the second detection frames based on the determination result.

[0048] Step 6: If there is at least one first detection frame, obtain a camera identifier corresponding to the current video image;

[0049] If one or more category labels are human labels, it can be determined that one or more first detection frames exist in all second detection frames, and the camera identifier of the camera that captures the current video image can be obtained.

[0050] Step seven: if there is no at least one first detection frame, determine that the target in each second detection frame is a static object in a sleeping posture.

[0051] If there is at least a part of the targets in the second detection frame whose corresponding category labels are not human labels, the targets in this part of the second detection frame can be considered to be static objects in a sleeping posture. If the category labels corresponding to the targets in each second detection frame do not contain at least one first detection frame, that is, the category labels corresponding to the targets in each second detection frame are not human labels, the targets in each second detection frame can be considered to be static objects in a sleeping posture.

[0052] Step eight, based on the camera identification, determine whether there is historical time information and at least one historical detection frame of the previous frame of video image corresponding to the camera identification; wherein the category label corresponding to the target in each historical detection frame is a human label, and the corresponding behavior label is a label associated with sleeping behavior.

[0053] In actual implementation, the historical time information of the previous frame of video image and at least one historical detection frame are usually saved in a database, such as redis, masql, etc., among which redis is an open source log-type, Key-Value database written in ANSI C language, supporting network, memory-based and persistent; mysql is an open source relational database management system; after obtaining the camera identifier, the database can be accessed to determine whether the historical time information of the previous frame of video image and at least one historical detection frame exist in the database.

[0054] Step nine, if there is historical time information and at least one historical detection frame, calculate a first time interval according to the historical time information and current time information of the current video image;

[0055] If the database actually contains historical time information of the previous frame of video image and at least one historical detection frame, the historical time information and at least one historical detection frame can be obtained, and the first time interval between the previous frame of video image and the current video image can be calculated based on the historical time information and the current time information of the current video image.

[0056] Step 10: if the first time interval is not greater than a preset time interval threshold, traverse each first detection frame and calculate the intersection-and-union ratio of each first detection frame with each historical detection frame;

[0057] The above-mentioned time interval threshold can be set according to actual needs; in actual implementation, after the above-mentioned first time interval is calculated, the first time interval can be compared with the preset time interval threshold to determine whether the first time interval is greater than the time interval threshold. If the first time interval is not greater than the time interval threshold, each first detection frame corresponding to the current video image can be traversed to calculate the intersection over union (IOU) of each first detection frame and each historical detection frame.

[0058] The following is a brief explanation of the concept of IOU: The calculation of IOU involves two key steps: first, determine the intersection area of ​​the predicted bounding box and the true bounding box, and calculate its area; second, determine the union area of ​​the two bounding boxes and calculate its area. The IOU value is the ratio of the intersection area to the union area, which reflects the degree of overlap between the predicted bounding box and the true bounding box.

[0059] Using mathematical formulas to express, the calculation of IOU is as follows:

[0060] IOU={Area(B_{pred}cap B_{gt})} / {Area(B_{pred}cup B_{gt})}

[0061] in:

[0062] -IOU is the ratio of intersection over union, which indicates the degree of overlap.

[0063] - B_{pred} is the predicted bounding box, usually output by the model.

[0064] -B_{gt} is the true bounding box, i.e., the bounding box in the annotated data.

[0065] -cap represents the intersection operation, which is used to calculate the overlapping area of ​​two bounding boxes.

[0066] -cup represents a union operation, which is used to calculate the total area covered by two bounding boxes.

[0067] -The Area function is used to calculate the area of ​​a region.

[0068] By calculating this ratio, we can get an IOU value between 0 and 1, which intuitively reflects the degree of overlap between the predicted bounding box and the true bounding box, and is therefore used to evaluate the performance of the target detection algorithm.

[0069] Step 11, based on the intersection-union ratio, using the DHash algorithm to detect whether there is post-sleeping behavior;

[0070] The step 11 is implemented by following the steps 110 to 112:

[0071] Step 110: if there is at least one detection frame pair whose IoU ratio is greater than a preset IoU ratio threshold, determine whether the first time interval is greater than a preset alarm time threshold;

[0072] The above-mentioned preset intersection-and-union ratio threshold can be set according to actual needs, such as 0.5; the above-mentioned alarm time threshold can be set according to actual needs; in actual implementation, by calculating the intersection-and-union ratio of each first detection frame and each historical detection frame, it is determined whether there is at least one detection frame pair whose intersection-and-union ratio is greater than the preset intersection-and-union ratio threshold, and the detection frame pair includes a first detection frame and a historical detection frame, and the intersection-and-union ratio of this first detection frame and this historical detection frame is greater than the preset intersection-and-union ratio threshold; if there is at least one detection frame pair, it is possible to continue to determine whether the first time interval is greater than the above-mentioned alarm time threshold.

[0073] Step 111: if the first time interval is greater than the alarm time threshold, for each detection frame pair, the similarity between the target in the first detection frame of the target and the target in the historical detection frame of the target is calculated by using the DHash algorithm;

[0074] For ease of understanding, the concept of the DHash algorithm is briefly explained below: DHash (Difference Hash) is a hash mapping method based on the pixel level. Image difference hashing is to compare the difference between the pixel values ​​of adjacent pixels pixel by pixel, that is, to obtain the difference between the current pixel and the right neighbor pixel pixel by pixel, to obtain an image difference matrix, and to generate a hash value through the matrix mapping. This pixel-by-pixel image difference matrix has one less column than the original pixel matrix, that is, the width is 1 less than the original pixel matrix, and the height remains unchanged, such as Figure 2 The flowchart of a DHash algorithm for generating hash values ​​is shown in FIG. 1 ; the steps for comparing image similarity are as follows:

[0075] 1. Scale the image to 9*8 pixels, that is, there are 8 rows and 9 pixels in each row;

[0076] 2. Convert the scaled image to a grayscale image, where each pixel is represented by an integer from 0 to 255;

[0077] 3. Compare each grayscale value with the grayscale value on its right row by row. If it is greater than, it is represented by 1, otherwise it is represented by 0. The grayscale values ​​between different rows are not compared. The result of the comparison can get 64 values ​​composed of 1 and 0. These 64 values ​​can be regarded as the fingerprint of the image.

[0078] 4. Compare the fingerprints of the two images, that is, compare the values ​​of the same position of the two fingerprints. If they are the same, no count is given. If they are different, count 1. Finally, if the count is less than sim_th, they are considered to be the same or similar pictures. Otherwise, they are considered to be two different pictures. Sim_th can be set according to the specific business.

[0079] In actual implementation, the DHash algorithm can be used to calculate the similarity between the target in the first detection frame and the target in the historical detection frame of the target in each detection frame pair.

[0080] Step 112, based on the similarity result, determine whether there is any post sleeping behavior.

[0081] The step 112 may be implemented by following steps A to B:

[0082] Step A: if the similarity is less than a preset similarity threshold, confirm that the target in the first target detection frame and the target in the target history detection frame are the same target, determine that there is a post sleeping behavior, and generate a post sleeping behavior alarm information;

[0083] The above-mentioned similarity threshold can be set according to actual needs; in actual implementation, the similarity between the target in the target first detection frame and the target in the target history detection frame in each calculated detection frame pair can be compared with the above-mentioned similarity threshold. For each detection frame pair, if the similarity between the target in the target first detection frame and the target in the target history detection frame in the detection frame pair is less than the similarity threshold, it can be considered that the target in the target first detection frame and the target in the target history detection frame are the same target, and it is determined that the target has a sleeping behavior. At this time, a sleeping behavior alarm information can be generated, and the sleeping behavior alarm information can be sent to a client, such as a PC front end, an APP mobile phone end, etc., or it can be prompted by sound alarms, indicator light alarms, etc., so as to promptly remind the user that the sleeping behavior is detected, so that the user can take corresponding measures in time.

[0084] Step B: If the similarity is not less than the similarity threshold, it is determined that there is no post sleeping behavior.

[0085] For each detection frame pair, if in the detection frame pair, the similarity between the target in the first detection frame of the target and the target in the historical detection frame of the target is not less than the similarity threshold, it can be considered that the target in the first detection frame of the target and the target in the historical detection frame of the target are not the same target, and it can be considered that there is no sleeping on the post behavior.

[0086] Step 12: If there is no historical time information and at least one historical detection frame, initialize the time of the camera corresponding to the camera identifier to the current time information.

[0087] If the historical time information and at least one historical detection frame of the previous frame of video image do not actually exist in the database, the historical time information and at least one historical detection frame cannot be obtained. At this time, the time of the camera corresponding to the camera identifier can be initialized to the current time information, and each first detection frame of the current video image can be saved in the database.

[0088] Step thirteen: if the first time interval is greater than a preset time interval threshold, it is determined that there is no post sleeping behavior.

[0089] Sleeping behavior usually has the following characteristics: the sleeper itself is a behavior that takes a long time and is in a static and stable posture. If the time interval between two detections of sleeping behavior by the same camera is large, it is largely not the behavior of the same target, and it is usually believed that the last detected sleeping behavior may be a false detection. It can also be understood that sleeping behavior is persistent and stable, so once it is detected, the next request should also be detected. The interval between two detections of sleeping behavior is the request interval, so the interval between two detections of sleeping behavior will not be too large. If the interval between two detections of sleeping behavior is too large, it can be considered not sleeping behavior.

[0090] In actual implementation, if the first time interval is greater than the time interval threshold, it is generally considered that the person having the sleeping behavior is not the same target and does not meet the characteristics of the sleeping behavior, and it can be determined that there is no sleeping behavior.

[0091] For easier understanding, see Figure 3 A system for detecting post sleeping behavior is shown, the system comprising: a user camera, a client (PC front end, APP mobile end), an AI (Artificial Intelligence) intelligent platform, and external connection components (kafka, redis, mysql) and other modules, wherein the AI ​​intelligent platform usually includes a gateway, through which post sleeping detection can be performed, and other detection functions such as fire monitoring can also be performed; the above-mentioned user camera is used to collect pictures or video data, and send the collected pictures or video data to kafka, and then send it to the AI ​​intelligent platform through kafka, and the AI ​​intelligent platform is used to analyze according to the method in the above-mentioned embodiment. If it is determined through analysis that there is post sleeping behavior, then post sleeping behavior alarm information can be generated, and the generated post sleeping behavior alarm information can be pushed to the client, and the user can also set relevant alarm parameters (such as inspection cycle, alarm frequency, alarm trigger conditions, etc.) through the client, and redis and mysql can be used to save relevant data, etc.

[0092] See also Figure 4 The flowchart of another method for detecting sleeping guard behavior is shown, where od_sleep is the AI ​​sub-capability for sleeping guard detection, which can be understood as a function that can perform sleeping guard detection; od_person is the AI ​​sub-capability for pedestrian detection, which can be understood as a function that can perform pedestrian detection; TIME represents the time interval, time represents the current moment, box is the target detection box, video_id is the camera identifier, which is used to avoid confusion between different cameras, and sim_th is the image comparison similarity threshold; the detection process is described below:

[0093] Step 1: Use od_sleep to detect the current video image, filter the sleep-related labels from the detection results, and determine whether there is a sleep target (corresponding to the above determination of whether there is a target behavior label associated with sleeping behavior in the current video image). If so, crop the corresponding coordinate frame position, that is, map and filter out the targets with sleeping behavior, and through image operations, crop the corresponding area according to the target's coordinate frame (corresponding to the above if there is at least one target behavior label, crop the image area where the target corresponding to each target behavior label is located), and then execute the next step; if not, directly return data as empty;

[0094] Step 2: For the cropped frame (corresponding to the second detection frame mentioned above), asynchronously request od_person to determine whether the sleeping target is of the "person" category (corresponding to the above-mentioned determination of whether there is at least one first detection frame with a category label of human according to the category label corresponding to the target in each second detection frame). If not, the detection target is a static object with a sleeping posture. In this case, data is directly returned as empty, and the time of the current video_id is reset to the current time, and the box information is set to empty; if so, proceed to the next step;

[0095] Step 3: For the current video image, obtain the video_id from the request parameters and access redis; obtain the time and box information of the previous video image corresponding to the video_id. If the acquisition is successful, it means that the video_id exists in redis, and then execute the next step; otherwise, initialize the time of the current video_id to the current time, insert the box information, and return data as empty;

[0096] Step 4: Calculate the time interval TIME between the previous video frame and the current video image, and determine whether the time interval TIME is greater than the maximum time interval threshold (corresponding to the above time interval threshold). If so, it means that they are not the same target to a great extent and do not meet the characteristics of sleeping. Time needs to be reset to the current time, the box information is updated, and data is returned as empty; otherwise, proceed to the next step;

[0097] Step 5: Traverse the box of the current video image (corresponding to the first detection box mentioned above), calculate the IOU with the historical box with the same video_id in redis (corresponding to the historical detection box mentioned above), and determine whether there is at least one detection box pair with an IOU greater than the threshold of 0.5. If so, execute the next step; otherwise, reset time to the current time, update the box information, and return data as empty;

[0098] Step 6: Determine whether the time interval TIME between the previous video image and the current video image is greater than the maximum alarm time threshold. For each detection frame pair, calculate the similarity between the target in the first detection frame and the target in the target history detection frame in the detection frame pair through the DHash algorithm, and compare the calculated similarity with the threshold sim_th. If it is less than sim_th, it is considered to be the same target. At this time, an alarm can be issued to indicate the occurrence of the post sleeping behavior, and the time information and box information are updated; otherwise, data is directly returned as empty;

[0099] The above-mentioned method for detecting sleeping on the post behavior, according to the characteristics of sleeping behavior, proposes a method that combines deep learning target detection technology, IOU comparison of previous and next frames, cumulative alarm time of sleeping, and Dhash similarity algorithm to comprehensively determine whether sleeping on the post behavior occurs or not; according to the characteristics of sleeping behavior, this method proposes time interval threshold and alarm time threshold, and determines sleeping on the post by combining multiple frames before and after; and according to the characteristics of sleeping behavior, the IOU of sleeping targets in previous and next frames is judged to determine whether they are the same target, thereby reducing false detection; this method also proposes to use DHash algorithm to calculate whether sleeping targets in previous and next frames are similar, so as to determine whether they are the same sleeping target; this method of using DHash for similarity judgment instead of convolutional neural network ReID (Person re-identification) can greatly improve the accuracy of sleeping on the post behavior detection under the premise of ensuring low latency and low resource loss of the overall process; at the same time, according to the actual business scenario, some parameters can be adjusted to control the recall rate of the overall business application, which has strong flexibility. By adopting the above-mentioned sleeping on the post behavior detection method, the decline of work efficiency and the occurrence of enterprise safety hazards can be effectively prevented.

[0100] The embodiment of the present invention provides a device for detecting post sleeping behavior. Figure 5 As shown, the device includes: a first acquisition module 50, which is used to acquire the video to be detected; a first detection module 51, which is used to detect the current video image for each frame of video image in the video to be detected, and obtain the detection result corresponding to the current video image; wherein the detection result includes: the category label and behavior label corresponding to each target in the current video image; a second acquisition module 52, which is used to obtain the camera identification corresponding to the current video image if it is determined according to the detection result that there is at least one first detection frame; wherein the category label corresponding to the target in each first detection frame is a human label, and the corresponding behavior label is a label associated with sleeping behavior; a second detection module 53, which is used to detect whether there is sleeping behavior based on the current time information, historical time information, each first detection frame and each historical detection frame of the current video image through the DHash algorithm if it is determined according to the camera identification that there is historical time information of the previous frame of video image corresponding to the camera identification and at least one historical detection frame; wherein the category label corresponding to the target in each historical detection frame is a human label, and the corresponding behavior label is a label associated with sleeping behavior.

[0101] The above-mentioned post sleeping behavior detection device can combine the current time information of the current video image, each first detection frame, and the historical time information of the previous frame of the video image and each historical detection frame, and use the DHash algorithm to detect whether there is post sleeping behavior, which can improve the detection accuracy of post sleeping behavior.

[0102] Furthermore, the second acquisition module is also used to: determine whether there is a target behavior label associated with sleeping behavior in the current video image based on the detection results; if there is at least one target behavior label, crop the image area where the target corresponding to each target behavior label is located to obtain a second detection frame of the target corresponding to each target behavior label; based on the category label corresponding to the target in each second detection frame, determine whether there is at least one first detection frame whose category label is a human label; if there is at least one first detection frame, obtain the camera identifier corresponding to the current video image.

[0103] Furthermore, the device is also used to: if there is no at least one first detection frame, determine that the target in each second detection frame is a static object with a sleeping posture.

[0104] Furthermore, the second detection module is also used to: determine, based on the camera identifier, whether there is historical time information and at least one historical detection frame of the previous frame of video image corresponding to the camera identifier; if there is historical time information and at least one historical detection frame, calculate the first time interval based on the historical time information and the current time information of the current video image; if the first time interval is not greater than a preset time interval threshold, traverse each first detection frame and calculate the intersection-and-union ratio of each first detection frame and each historical detection frame; based on the intersection-and-union ratio, detect whether there is post-sleeping behavior through the DHash algorithm.

[0105] Furthermore, the device is also used to: if there is no historical time information and at least one historical detection frame, initialize the time of the camera corresponding to the camera identifier to the current time information.

[0106] Furthermore, the device is also used to: if the first time interval is greater than a preset time interval threshold, determine that there is no sleeping on duty behavior.

[0107] Furthermore, the second detection module is also used for: if there is at least one detection frame pair whose intersection-and-unit ratio is greater than a preset intersection-and-unit ratio threshold, determining whether the first time interval is greater than a preset alarm time threshold; if the first time interval is greater than the alarm time threshold, calculating, for each detection frame pair, the similarity between the target in the first detection frame of the target and the target in the historical detection frame of the target in the detection frame pair by using the DHash algorithm; and determining whether there is any post-sleeping behavior based on the similarity result.

[0108] Furthermore, based on the similarity result, the step of determining whether there is any sleeping on the post behavior includes: if the similarity is less than a preset similarity threshold, confirming that the target in the first target detection frame and the target in the target history detection frame are the same target, determining that there is any sleeping on the post behavior, and generating a sleeping on the post behavior alarm message; if the similarity is not less than the similarity threshold, determining that there is no sleeping on the post behavior.

[0109] The implementation principle and technical effects of the post sleeping behavior detection device provided in the embodiment of the present invention are the same as those of the aforementioned post sleeping behavior detection method embodiment. For the sake of brief description, for matters not mentioned in the embodiment of the post sleeping behavior detection device, reference may be made to the corresponding contents in the aforementioned post sleeping behavior detection method embodiment.

[0110] The embodiment of the present invention further provides an electronic device, see Figure 6 As shown, the electronic device includes a processor 130 and a memory 131. The memory 131 stores machine executable instructions that can be executed by the processor 130. The processor 130 executes the machine executable instructions to implement the above-mentioned post sleeping behavior detection method.

[0111] Further, Figure 6 The electronic device shown further includes a bus 132 and a communication interface 133 , and the processor 130 , the communication interface 133 and the memory 131 are connected via the bus 132 .

[0112] The memory 131 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 132 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0113] The processor 130 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 130. The above processor 130 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131 and completes the steps of the method of the above embodiment in combination with its hardware.

[0114] An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned post sleeping behavior detection method. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0115] The computer program product of the method, device and electronic device for detecting post sleeping behavior provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0116] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting post sleeping behavior, characterized in that: The method comprises: Get the video to be detected; For each frame of the video image in the video to be detected, the current video image is detected to obtain a detection result corresponding to the current video image; wherein the detection result includes: a category label and a behavior label corresponding to each target in the current video image; If it is determined according to the detection result that there is at least one first detection frame, a camera identifier corresponding to the current video image is obtained; wherein the category label corresponding to the target in each first detection frame is a human label, and the corresponding behavior label is a label associated with sleeping behavior; If it is determined based on the camera identification that there is historical time information and at least one historical detection frame of the previous frame of the video image corresponding to the camera identification, based on the current time information of the current video image, the historical time information, each of the first detection frames and each of the historical detection frames, the DHash algorithm is used to detect whether there is sleeping on the post behavior; wherein the category label corresponding to the target in each of the historical detection frames is the human label, and the corresponding behavior label is a label associated with sleeping behavior.

2. The method according to claim 1, characterized in that If it is determined according to the detection result that there is at least one first detection frame, the step of obtaining a camera identifier corresponding to the current video image includes: According to the detection result, determining whether there is a target behavior label associated with sleeping behavior in the current video image; If there is at least one target behavior label, cropping the image region where the target corresponding to each target behavior label is located to obtain a second detection frame of the target corresponding to each target behavior label; According to the category label corresponding to the target in each second detection frame, determining whether there is at least one first detection frame whose category label is a human label; If there is at least one of the first detection frames, a camera identifier corresponding to the current video image is obtained.

3. The method according to claim 2, characterized in that The method further comprises: If there is no at least one of the first detection frames, it is determined that the target in each of the second detection frames is a static object in a sleeping posture.

4. The method according to claim 1, characterized in that If it is determined according to the camera identifier that there is historical time information and at least one historical detection frame of the previous frame of the video image corresponding to the camera identifier, the step of detecting whether there is a post sleeping behavior by using a DHash algorithm based on the current time information of the current video image, the historical time information, each of the first detection frames and each of the historical detection frames includes: According to the camera identifier, determining whether there is historical time information and at least one historical detection frame of a previous frame of video image corresponding to the camera identifier; If the historical time information and at least one historical detection frame exist, calculating a first time interval according to the historical time information and current time information of the current video image; If the first time interval is not greater than a preset time interval threshold, traverse each of the first detection frames and calculate an intersection-over-union ratio between each of the first detection frames and each of the historical detection frames; Based on the intersection-union ratio, the DHash algorithm is used to detect whether there is post-sleeping behavior.

5. The method according to claim 4, characterized in that The method further comprises: If the historical time information and at least one historical detection frame do not exist, the time of the camera corresponding to the camera identifier is initialized to the current time information.

6. The method according to claim 4, characterized in that The method further comprises: If the first time interval is greater than a preset time interval threshold, it is determined that there is no post sleeping behavior.

7. The method according to claim 4, characterized in that Based on the intersection-and-union ratio, the steps of detecting whether there is post-sleeping behavior by using the DHash algorithm include: If there is at least one detection frame pair whose IoU ratio is greater than a preset IoU threshold, determining whether the first time interval is greater than a preset alarm time threshold; If the first time interval is greater than the alarm time threshold, for each detection frame pair, calculating the similarity between the target in the first detection frame of the target and the target in the historical detection frame of the target in the detection frame pair by using the DHash algorithm; Based on the similarity result, determine whether there is post-sleeping behavior.

8. The method according to claim 7, characterized in that According to the similarity result, the steps of determining whether there is a post sleeping behavior include: If the similarity is less than a preset similarity threshold, confirm that the target in the first target detection frame and the target in the target history detection frame are the same target, determine that there is a post-sleeping behavior, and generate post-sleeping behavior alarm information; If the similarity is not less than the similarity threshold, it is determined that the post-sleeping behavior does not exist.

9. A device for detecting post sleeping behavior, characterized in that: The device comprises: A first acquisition module is used to acquire a video to be detected; A first detection module is used to detect the current video image for each frame of the video image to be detected, and obtain a detection result corresponding to the current video image; wherein the detection result includes: a category label and a behavior label corresponding to each target in the current video image; A second acquisition module is configured to acquire a camera identifier corresponding to the current video image if it is determined that there is at least one first detection frame according to the detection result; wherein the category label corresponding to the target in each first detection frame is a human label, and the corresponding behavior label is a label associated with sleeping behavior; The second detection module is used to determine, based on the camera identification, the existence of historical time information and at least one historical detection frame of the previous frame of the video image corresponding to the camera identification, and detect whether there is sleeping on the post through the DHash algorithm based on the current time information of the current video image, the historical time information, each of the first detection frames and each of the historical detection frames; wherein the category label corresponding to the target in each of the historical detection frames is the human label, and the corresponding behavior label is a label associated with sleeping behavior.

10. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the post sleeping behavior detection method according to any one of claims 1-8.

Images