Method, system and storage medium for detecting employee sleepiness based on artificial intelligence

Abstract

The application discloses a method and system for detecting sleep-on-duty of employees based on artificial intelligence and a storage medium. The method comprises the following steps: obtaining picture data to be reasoned based on a set time period according to a current time from a monitoring image; calculating positions of effective human body key points at each time in the set time period according to the picture data to be reasoned; calculating distance differences between positions of each effective human body key point at each time and positions of each effective human body key point at each adjacent time; and determining that a person has a sleep-on-duty behavior if the distance differences between the positions of each effective human body key point at each time and the positions of each effective human body key point at each adjacent time are all less than a distance threshold. The method can determine whether a value-on-duty person has a sleep-on-duty behavior through artificial intelligence. The application can realize 24-hour monitoring of sleep-on-duty through artificial intelligence, reduce the cost of employee management, achieve the purpose of cost reduction and efficiency improvement, improve the service quality of employees on duty, and avoid safety hazards.

Description

Technical Field

[0001] The present invention relates to the field of property monitoring technology, and in particular to a method, system and storage medium for detecting employees sleeping on duty based on artificial intelligence. Background Art

[0002] With the continuous development of enterprises, the requirements for property management services are becoming increasingly higher. However, in property management, employees often sleep on the job. This is not only a dereliction of duty for property management staff, but also a serious safety hazard that cannot be ignored. For example, monitoring room personnel may miss some key surveillance images, resulting in safety risks for property owners.

[0003] In real life, property management companies have formulated a series of industry service standards for property management personnel, and have used a series of methods such as offline inspections, offline patrols, offline watermark photography and clocking in, and online video monitoring of property management personnel to solve the problems of property management personnel on the job and whether there is any behavior of sleeping on the job. For example, through random and uninterrupted spot checks on employees' on-the-job status, such dereliction of duty as sleeping on the job can be reduced, thus avoiding some unnecessary safety accidents.

[0004] However, in the traditional method of controlling property staff sleeping on the job, whether it is supervision by supervisors or auxiliary supervision by third-party supervision software, such as photo clocking-in systems, video surveillance spot-check service post software, etc., it leads to high labor costs and low processing efficiency. Summary of the Invention

[0005] In view of the above technical problems, the purpose of the present invention is to provide a method, system and storage medium for detecting employee sleeping on the job based on artificial intelligence, so as to solve the problems of high labor costs and low processing efficiency in traditional methods of monitoring property personnel sleeping on the job.

[0006] The present invention adopts the following technical solutions:

[0007] The method for detecting employees sleeping on the job based on artificial intelligence includes the following steps:

[0008] According to the monitoring image, the image data to be inferred in the set time period based on the current time is obtained;

[0009] Calculate the positions of the effective human body key points at each moment within the set time period based on the image data to be inferred;

[0010] According to the position of each effective human body key point at each moment, the distance difference between the positions of each effective human body key point at each two adjacent moments is calculated;

[0011] If the distance difference between the positions of each valid human key point at each two adjacent moments is less than the distance threshold, it is determined that the person is sleeping on the job.

[0012] Preferably, obtaining the image data to be inferred for a set time period based on the current time according to the monitoring image includes:

[0013] The video stream of the monitoring device in the set time period is obtained, and the video stream is framed to obtain image data. The detection area is marked on the image data, and then the marked detection area is image cut. The cut image data is then scaled proportionally to the set standard image size to obtain the image data to be inferred in the set time period.

[0014] Preferably, the calculating the positions of the effective human body key points at each moment in the set time period based on the image data to be inferred includes:

[0015] Input the image data to be inferred into a deep learning model, and output a number of human key points and the confidence level of each human key point;

[0016] The human body key points corresponding to the human body key points whose confidence is greater than or equal to the confidence threshold are taken as valid human body key points, and the positions of the valid human body key points at each moment within the set time period are calculated.

[0017] Preferably, the key points of the human body include but are not limited to one or more of nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist.

[0018] Preferably, the time intervals between every two adjacent moments in the set time period are the same.

[0019] Preferably, the distance difference between the positions of the key points of the human body at every two adjacent moments satisfies the following formula:

[0020]

[0021] Among them, n is the number of valid key points, t m-1 With t m are two adjacent moments;

[0022] t m The distance difference between the positions of the nth valid key points at the moment, t m The horizontal distance of the position of the nth valid key point at the moment, t m-1 The horizontal distance of the position of the nth valid key point at the moment, t m The vertical distance of the position of the nth valid key point at the moment, t m-1 The vertical distance of the position of the nth valid key point at the moment.

[0023] The AI-based system for detecting employees sleeping on the job includes:

[0024] An image acquisition unit is used to obtain image data to be inferred within a set time period based on the current time according to the monitoring image;

[0025] A position calculation unit, configured to calculate the positions of valid human body key points at each moment within the set time period based on the image data to be inferred;

[0026] A distance calculation unit is used to calculate the distance difference between the positions of each valid human body key point at each two adjacent moments according to the position of each valid human body key point at each moment;

[0027] The determination unit is used to determine that a person is sleeping on duty if the distance difference between the positions of each valid human body key point at each two adjacent moments is less than a distance threshold.

[0028] Preferably, the image acquisition unit includes an image preprocessing unit, a human body key point reasoning unit and a confidence screening calculation unit;

[0029] The image preprocessing unit is used to obtain the video stream of the monitoring device in a set time period, extract frames from the video stream to obtain image data, mark the detection area on the image data, then perform image segmentation on the marked detection area, and then perform geometric scaling on the segmented image data to a set standard image size to obtain the image data to be inferred in the set time period;

[0030] The human body key point inference unit is used to input the image data to be inferred into a deep learning model and output a number of human body key points and the confidence level of each human body key point;

[0031] The confidence screening calculation unit is used to take the human body key points corresponding to the human body key points whose confidence is greater than or equal to the confidence threshold as valid human body key points, and calculate the positions of the valid human body key points at each moment within the set time period.

[0032] An electronic device comprises: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for detecting employees sleeping on the job based on artificial intelligence.

[0033] A computer storage medium stores a computer program, which, when executed by a processor, implements the method for detecting employee sleep-on-the-job based on artificial intelligence.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention obtains image data to be inferred for a set time period based on the current time based on surveillance images; calculates the positions of valid human key points at each moment in the set time period based on the image data to be inferred; calculates the distance difference between the positions of each valid human key point at each two adjacent moments based on the positions of each valid human key point at each moment; if the distance difference between the positions of each valid human key point at each two adjacent moments is less than a distance threshold, that is, the fluctuation of the upper body movement of the human body within the continuous time of the set time period is less than a reference value, then it is determined that the person has engaged in sleeping on duty. This method uses artificial intelligence to infer whether the on-duty personnel have engaged in sleeping on duty, for example, sleeping on a work table, sleeping leaning on a chair, sleeping on a stool, sleeping with their eyes closed while standing guard, etc., or whether they have slept on duty for more than a set time, thereby outputting a sleeping on duty behavior alarm event with an accuracy rate of over 90%. This method can control the phenomenon of on-duty personnel dereliction of duty in property services and avoid safety hazards. At the same time, artificial intelligence can be used to monitor sleeping on duty 24 hours a day, which can reduce management personnel costs, achieve the goal of reducing costs and increasing efficiency, and improve the quality of on-duty service provided by employees. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a method for detecting employees sleeping on the job based on artificial intelligence provided by one embodiment of the present invention;

[0037] Figure 2 A schematic diagram of a method for detecting employees sleeping on the job based on artificial intelligence according to a specific embodiment of the present invention;

[0038] Figure 3 A schematic flow chart of a method for identifying human motion according to a specific embodiment of the present invention;

[0039] Figure 4 A schematic diagram of a system for detecting employee sleep-on-duty based on artificial intelligence provided by one embodiment of the present invention;

[0040] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be noted that, without conflict, the embodiments or technical features described below may be arbitrarily combined to form new embodiments.

[0042] Example 1:

[0043] Please refer to Figure 1-5 As shown, Figure 1The method for detecting employee sleeping on duty based on artificial intelligence is shown, comprising the following steps:

[0044] Step S1: obtaining picture data to be inferred of a set time period based on current time according to a monitoring image;

[0045] Preferably, the step S1 specifically comprises:

[0046] acquiring a video stream of a monitoring device in a set time period, and frame extracting the video stream to obtain picture data, labeling a detection area of the picture data, then image cutting the labeled detection area, and equi-scaling the cut image data to a set standard picture size to obtain picture data to be inferred of the set time period.

[0047] Specifically, the picture data is acquired from a non-AI digital camera and an analog camera; specifically, a variety of protocol cameras on the market are connected through a video connection unit, and the video stream of the corresponding device can be acquired through the unit; the video connection unit has the characteristics of simultaneously connecting and being compatible with cameras of multiple video security manufacturers; then the video stream is frame extracted through a video frame extraction unit, so that the stream data in the video stream is converted into picture data; and the video frame extraction unit is dynamically configured with rules from an external interface to adjust the frame extraction strategy.

[0048] Specifically, the detection area is acquired through a detection area configuration unit, and specifically, the detection area is pre-labeled by a business staff through a Web visual method.

[0049] Specifically, the detection area in the detection area configuration unit is identified and cut through a picture identification cutting unit, and if the upper left corner of the original picture is set as (0, 0) and the lower right corner is (w, h), where w is the width of the original picture and h is the height of the original picture; the upper left corner coordinate of the rectangular area is (w1, h1) and the lower right corner coordinate is (w2, h2), forming a rectangular area with (w1, h1), (w1, h2), (w2, h2), and (w2, h1) as the connecting lines, and the area is cut to obtain the cut picture.

[0050] In this embodiment, the detection area is a rectangular area set by a human being.

[0051] Step S2: calculating the positions of effective human body key points at each time in the set time period according to the picture data to be inferred;

[0052] In this embodiment, the current position state of the human body is acquired through an artificial intelligence algorithm, the human body skeleton contour in the image is depicted, and the positions of the current human body key points are determined.

[0053] In this embodiment, the artificial intelligence algorithm uses an artificial intelligence reasoning model as the PoseNet deep learning model.

[0054] The input format of the model is the input pixel value and number of channels specified by the artificial intelligence reasoning model.

[0055] For example, the input of the algorithm model is 224×224×3, where the first two values ​​are length and width respectively, and the third is the number of image channels. Using a rectangular monitoring area can achieve the specified input effect through regular padding and scaling.

[0056] When performing image recognition through the artificial intelligence inference model, 17 key points of the human body will be obtained, namely the left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle, etc. The inference results are output to obtain the coordinate position and confidence of each key.

[0057] The confidence level is a value between 0 and 1, and the closer to 1, the more reliable it is.

[0058] Preferably, the step S2 includes:

[0059] Input the image data to be inferred into a deep learning model, and output a number of human key points and the confidence level of each human key point;

[0060] The human body key points corresponding to the human body key points whose confidence is greater than or equal to the confidence threshold are taken as valid human body key points, and the positions of the valid human body key points at each moment within the set time period are calculated.

[0061] Step S3: According to the position of each effective human body key point at each moment, calculate the distance difference between the positions of each effective human body key point at every two adjacent moments;

[0062] Preferably, the key points of the human body include but are not limited to one or more of nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist.

[0063] Step S4: If the distance difference between the positions of each valid human key point at each two adjacent moments is less than the distance threshold, it is determined that the person is sleeping on duty.

[0064] Preferably, the time intervals between every two adjacent moments in the set time period are the same.

[0065] Preferably, the distance difference between the positions of the key points of the human body at every two adjacent moments satisfies the following formula:

[0066]

[0067] Among them, n is the number of valid key points, tm-1 With t m are two adjacent moments;

[0068] t m The distance difference between the positions of the nth valid key points at the moment, t m The horizontal distance of the position of the nth valid key point at the moment, t m-1 The horizontal distance of the position of the nth valid key point at the moment, t m The vertical distance of the position of the nth valid key point at the moment, t m-1 The vertical distance of the position of the nth valid key point at the moment.

[0069] In a specific implementation, the key points of the upper body of the human body include 11 positions, namely nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist, which are numbered from 0 to 10. Among them, each number represents a key point.

[0070] In the above implementation process, the present invention determines whether the human body has moved through the key points of the human body, can repeatedly infer the key points of the human body within a period of time, and judge the movement of the human body through the displacement of the current and historical key points, thereby judging the sleeping behavior.

[0071] The method of the present invention is described below with specific embodiments:

[0072] Please refer to Figure 2 As shown, Figure 2 The present invention shows a method for detecting employees sleeping on the job based on artificial intelligence. It determines whether the human body has moved by the key points of the human body. It can repeatedly infer the key points of the human body over a period of time and determine the movement of the human body by the displacement of the current and historical key points, thereby determining the sleeping behavior.

[0073] Specifically, the system for calculating point distances through a time window is used for discrimination. The system for calculating point distances through a time window specifically includes a window calculation unit, a key point distance calculation unit, a human motion discrimination unit, and a data cache unit. The window calculation unit is responsible for collecting the key points of the inference results and extracting the historical key point data from the data cache unit. The key point distance unit calculates the distance between the key points corresponding to the historical image and the current image.

[0074] The human motion identification unit is responsible for error analysis of the movement distance of key points at each time point in the window, and then judges the sleeping behavior. Please refer to Figure 3 As stated, Figure 3 The specific method of human motion identification of the present invention is shown, and the steps specifically include:

[0075] 1. Obtain a continuous time frame picture data set;

[0076] 2. Input a human key point detection model to obtain human key point recognition;

[0077] 3. The continuous time frame has small human body movement fluctuation;

[0078] 4. Trigger a sleep behavior alarm.

[0079] In the embodiment, the window is a time window, and a time period is defined in the window. The pictures input in the time period are calculated for the window.

[0080] Specifically, assuming that the current time is t m , the coordinates of the key points 0-10 are respectively

[0081] Obtain the t m-1 time key points from the cache, which are respectively

[0082] Calculate the distance of the key point n corresponding to the t m-1 to t m time period

[0083]

[0084] When is less than the reference value a, it is considered that the human body does not move in the t m-1 to t m time period;

[0085] The window time period is cut into m segments. When each segment is determined to not move, the current human body sleep behavior is determined.

[0086] In the above implementation process, by monitoring the image, the to-be-reasoned picture data of the set time period based on the current time is obtained; the positions of the effective human key points at each time in the set time period are calculated according to the to-be-reasoned picture data; the distance difference of each effective human key point at each time is calculated according to the positions of each effective human key point at each time; if the distance difference of each effective human key point at each time is less than the distance threshold, that is, the fluctuation of the upper body movement of the human body in the continuous time of the set time period is less than the reference value, it is determined that the personnel has a sleep behavior, the artificial intelligence method is used to reason and determine whether the value guard personnel has a sleep behavior, for example, whether the sleep behavior exceeds the set time, so as to output a sleep behavior alarm event, the accuracy rate reaches more than 90%, the value guard personnel in the property service can be controlled to avoid the occurrence of security risks.

[0087] Example 2:

[0088] Please refer to Figure 4 As shown, Figure 4 The present invention shows a system for detecting employees sleeping on the job based on artificial intelligence, comprising:

[0089] An image acquisition unit is used to obtain image data to be inferred within a set time period based on the current time according to the monitoring image;

[0090] A position calculation unit, configured to calculate the positions of valid human body key points at each moment within the set time period based on the image data to be inferred;

[0091] A distance calculation unit is used to calculate the distance difference between the positions of each valid human body key point at each two adjacent moments according to the position of each valid human body key point at each moment;

[0092] The determination unit is used to determine that a person is sleeping on duty if the distance difference between the positions of each valid human body key point at each two adjacent moments is less than a distance threshold.

[0093] Preferably, the image acquisition unit includes an image preprocessing unit, a human body key point reasoning unit and a confidence screening calculation unit;

[0094] The image preprocessing unit is used to obtain the video stream of the monitoring device in a set time period, extract frames from the video stream to obtain image data, mark the detection area on the image data, then perform image segmentation on the marked detection area, and then perform geometric scaling on the segmented image data to a set standard image size to obtain the image data to be inferred in the set time period;

[0095] The human body key point inference unit is used to input the image data to be inferred into a deep learning model and output a number of human body key points and the confidence level of each human body key point;

[0096] The confidence screening calculation unit is used to take the human body key points corresponding to the human body key points whose confidence is greater than or equal to the confidence threshold as valid human body key points, and calculate the positions of the valid human body key points at each moment within the set time period.

[0097] Example 3:

[0098] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 The schematic diagram shown is used to describe the electronic device 100 used to implement the method for detecting employees sleeping on the job based on artificial intelligence according to an embodiment of the present application.

[0099] like Figure 5 The electronic device 100 includes one or more processors 102 and one or more storage devices 104. These components are interconnected via a bus system and / or other forms of connection mechanisms (not shown). Figure 5 The components and structure of the electronic device 100 shown are only exemplary and not restrictive. The electronic device may have Figure 5 Some of the components shown may also have Figure 5 Other components and structures are not shown.

[0100] The processor 102 may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0101] The storage device 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the functions (implemented by the processor) in the embodiments of the present application described below and / or other desired functions. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the application.

[0102] The application further provides a computer storage medium, which stores a computer program. The method of the application can be stored in the computer storage medium if the method is implemented in the form of a software function unit and sold or used as an independent product. Based on this understanding, the application can implement all or part of the processes in the above-mentioned method embodiments, and can also be completed by instructing related hardware through a computer program. The computer program can be stored in a computer storage medium. When the computer program is executed by a processor, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form, etc. The computer storage medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in a jurisdiction. For example, according to legislation and patent practice in some jurisdictions, the computer storage medium does not include an electrical carrier signal and a telecommunication signal.

[0103] For those skilled in the art, various corresponding changes and modifications can be made to the technical solutions and concepts described above, and all these changes and modifications should belong to the protection scope of the claims of the application.

Claims

1. A method for detecting employees sleeping on the job based on artificial intelligence, characterized in that: The following steps are involved: According to the monitoring image, the image data to be inferred in the set time period based on the current time is obtained; Calculate the positions of the effective human body key points at each moment within the set time period based on the image data to be inferred; Calculating the positions of valid human body key points at each moment within the set time period based on the image data to be inferred includes: Input the image data to be inferred into a deep learning model, and output a number of human key points and the confidence level of each human key point; The human body key points corresponding to the human body key points whose confidence levels are greater than or equal to the confidence threshold are taken as valid human body key points, and the positions of the valid human body key points at each moment within the set time period are calculated; When performing image recognition through the artificial intelligence inference model, 17 key points of the human body are obtained, namely the left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. According to the position of each effective human body key point at each moment, the distance difference between the positions of each effective human body key point at each two adjacent moments is calculated; If the distance difference between the positions of each valid human key point at each two adjacent moments is less than the distance threshold, it is determined that the person is sleeping on duty; The step of obtaining image data to be inferred for a set time period based on the current time according to the monitoring image includes: The video stream of the monitoring device in the set time period is obtained, and the video stream is framed to obtain image data. The detection area is marked on the image data, and then the marked detection area is image cut. The detection area in the detection area unit is configured through the image recognition and cutting unit, and the image is recognized and cut. The cut image data is then scaled proportionally to the set standard image size to obtain the image data to be inferred in the set time period.

2. The method for detecting employees sleeping on the job based on artificial intelligence according to claim 1, characterized in that: The time intervals between every two adjacent moments in the set time period are the same.

3. The method for detecting employees sleeping on the job based on artificial intelligence according to claim 1, characterized in that: The distance difference between the positions of the key points of the human body at each two adjacent moments satisfies the following formula: Among them, n is the number of valid key points, t m-1 With t m are two adjacent moments; t m The distance difference between the positions of the nth valid key points at the moment, t m The horizontal distance of the position of the nth valid key point at the moment, t m-1 The horizontal distance of the position of the nth valid key point at the moment, t m The vertical distance of the position of the nth valid key point at the moment, t m-1 The vertical distance of the position of the nth valid key point at the moment.

4. A system based on artificial intelligence to detect employees sleeping on the job, characterized by: include: An image acquisition unit is used to obtain image data to be inferred within a set time period based on the current time according to the monitoring image; A position calculation unit, configured to calculate the positions of valid human body key points at each moment within the set time period based on the image data to be inferred; The method comprises: inputting the image data to be inferred into a deep learning model, outputting a number of human key points and the confidence of each human key point; taking the human key points corresponding to the human key point confidence greater than or equal to the confidence threshold as the valid human key points, and calculating the positions of the valid human key points at each moment in the set time period; when performing image recognition through the artificial intelligence inference model, 17 human key points are obtained, namely, the positions of the left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle; A distance calculation unit is used to calculate the distance difference between the positions of each valid human body key point at each two adjacent moments according to the position of each valid human body key point at each moment; A determination unit is configured to determine that a person is sleeping on duty if the distance difference between the positions of each valid human key point at each two adjacent moments is less than a distance threshold; the image acquisition unit includes an image preprocessing unit, a human key point reasoning unit, and a confidence screening calculation unit; The image preprocessing unit is used to obtain the video stream of the monitoring device in a set time period, extract frames from the video stream to obtain image data, mark the detection area on the image data, and then perform image segmentation on the marked detection area. The image recognition and segmentation unit recognizes and segments the detection area in the detection area configuration unit, and then scales the segmented image data proportionally to a set standard image size to obtain the image data to be inferred for the set time period; The human body key point inference unit is used to input the image data to be inferred into a deep learning model and output a number of human body key points and the confidence level of each human body key point; The confidence screening calculation unit is used to take the human body key points corresponding to the human body key points whose confidence is greater than or equal to the confidence threshold as valid human body key points, and calculate the positions of the valid human body key points at each moment within the set time period.

5. An electronic device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for detecting employees sleeping on the job based on artificial intelligence as described in any one of claims 1 to 3.

6. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting employees sleeping on the job based on artificial intelligence according to any one of claims 1 to 3 is implemented.

Images