A mobile phone playing and off-duty detection alarm system and method based on deep learning

Through the deep learning-based mobile phone play and off-duty detection alarm system, the target detection technology is used to monitor employee behavior in real time, solving the problem of employees playing with mobile phones or off-duty during working hours, and achieving accurate and real-time supervision and management of employee behavior. Improve work efficiency.

CN115019236BActive Publication Date: 2024-11-08禾麦科技开发(深圳)有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210739096.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-27
Publication Date
2024-11-08
Estimated Expiration
2042-06-27

AI Technical Summary

Technical Problem

In a work environment, employees' behavior of playing with their mobile phones during working hours or leaving their jobs is difficult to be supervised in real time, resulting in reduced work efficiency and the possibility of managers turning a blind eye to the company, which harms the interests of the company.

Method used

Using deep learning-based mobile phone play and off-duty detection alarm system, video data is transmitted by installing a camera, Yolov5 target detection network and Resnet50 secondary classification network to detect human and mobile phone targets in real time, and combining location information to determine whether employees are playing on mobile phones or off-duty , and upload the results to the background management server for recording and alert notification.

Benefits of technology

Accurate and real-time supervision of employee behavior is achieved, and work efficiency is improved. Managers can monitor employees' working status in real time through browsers, reducing subjective deviations of managers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115019236B_ABST
    Figure CN115019236B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of image detection and recognition technology, and discloses a deep learning-based mobile phone playing and off-duty detection alarm system and method, including a monitoring camera, an image storage server and a background management server, wherein the monitoring camera is data-connected to the image storage server, and the image storage server is data-connected to the background management server respectively, and the image storage server is used to detect the human body and mobile phone targets in the image, and mark their location information, and then judge whether the employee is playing with the mobile phone or off-duty according to the location information of the human body and the mobile phone, and after detecting the behavior of the employee playing with the mobile phone or off-duty, the result is uploaded to the background management server, and the background management server is used to record the behavior and display it in real time, and to alert the administrator. The present invention automatically identifies whether the employee is playing with the mobile phone or off-duty, and can help managers to monitor the work situation of employees during working hours in real time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image detection and recognition, and in particular to a mobile phone playing and off-duty detection alarm system and method based on deep learning. Background Art

[0002] In daily work, some employees play with their mobile phones or leave their posts during working hours, which not only reduces work efficiency but also has a serious negative impact on the company's working environment. Usually, in order to deal with such problems, managers need to supervise. However, managers cannot supervise in real time, and there may be a phenomenon that managers turn a blind eye or even go along with the corrupt practices, which is detrimental to the interests of the company. Summary of the invention

[0003] The main purpose of this invention is to propose a deep learning-based mobile phone playing and off-duty detection alarm system and method, which aims to transmit real-time video data through a camera installed on the workstation desk or ceiling to complete the judgment of whether the employee is playing with the mobile phone or is off-duty, and help managers to monitor the work status of employees during working hours in real time.

[0004] To achieve the above-mentioned purpose, the present invention proposes a deep learning-based mobile phone playing and leaving the post detection alarm system, including monitoring cameras, image storage servers and background management servers. The monitoring cameras are respectively arranged above the office area, and the monitoring cameras are used to monitor the behavior of people in the office area. The monitoring cameras are data-connected to the image storage servers, and the image storage servers are data-connected to the background management servers respectively. The image storage server is used to detect human bodies and mobile phone targets in the image, and mark their location information, and then judge whether the employees are playing with mobile phones or leaving their posts based on the location information of the human bodies and mobile phones. After detecting the behavior of employees playing with mobile phones or leaving their posts, the results are uploaded to the background management server. The background management server is used to record the behavior and display it in real time, and notify the administrator of the alarm, so that the administrator can monitor the system operation status and the behavior of employees in the office area in real time through the browser.

[0005] The present invention also provides a method for detecting and alarming mobile phone playing and leaving the post based on deep learning, which is implemented by the above system and includes the following steps:

[0006] The camera and image algorithm server are connected to the same monitoring intranet;

[0007] The administrator starts the backend management server and logs in to the backend management website to configure the parameters of the image algorithm server and set the video stream address of the surveillance camera;

[0008] The administrator starts the image storage server from the backend management website;

[0009] After the camera transmits the monitoring image to the image storage server, the image storage server inputs the image into the target detection algorithm to detect the human body and mobile phone targets in the image in real time and mark their location information;

[0010] The image storage server determines whether an employee is playing with their phone or off-duty based on the location information of the human body and the mobile phone. If no human body is detected at an employee's workstation for more than a preset time threshold, the employee is considered to be off-duty. If a mobile phone appears in the human body frame marked by the target detection algorithm for more than a preset time threshold, the employee is considered to be playing with his phone.

[0011] After the image storage server detects that an employee is playing with his / her mobile phone or is away from work, it uploads the result to the background management server, which records the behavior and displays it in the form of a web page.

[0012] Managers can monitor employees' real-time and past work status through the backend management website.

[0013] Furthermore, if no human target is detected at a certain employee's workstation for more than a preset time threshold, then it is determined that the employee has left the post, including the following steps:

[0014] When the algorithm detects abnormal behavior in a certain frame, it starts timing and detects the next frames. When no abnormal behavior is detected in a certain frame, the timing does not end immediately. Instead, it ends when there is no abnormal behavior for N consecutive frames. Then, it determines whether it is a real abnormal behavior based on whether the duration of the abnormal behavior exceeds the preset time threshold.

[0015] Furthermore, the target detection algorithm detects human bodies and mobile phone targets in the image and marks their location information, including the following steps:

[0016] The camera is connected to the network through the Real-Time Streaming Protocol (RTSP). The video stream processing thread directly reads the video stream from the monitoring network and transmits it to the Yolov5 target detection network after processing. The video stream processing thread consists of a writing thread and a reading thread. The writing thread is responsible for reading the monitoring screen and writing it to the queue, and maintaining the queue size at 1 through the producer-consumer model; the reading thread reads the latest frame from the queue, and transmits it to the Yolov5 target detection network after preprocessing;

[0017] The Yolov5 target detection network obtains the latest frame from the reading thread, extracts image features through the convolutional neural network, and predicts the target position of the person and the mobile phone in the frame and the corresponding confidence. It also screens according to the confidence threshold, removes the results with low confidence, and obtains the preliminary prediction results;

[0018] After the Yolov5 target detection network detects the mobile phone target, the mobile phone target is intercepted and the Resnet50 secondary classification network is used to perform the first round of screening to filter out some misidentified targets;

[0019] For mobile phone targets that have been filtered by Resnet50, another round of screening is performed. First, Gaussian smoothing filtering is used to eliminate Gaussian noise, then the canny edge detection algorithm is used to obtain its edge features, and finally the Hough line detection algorithm is used to detect the number of straight lines in the edge features. If the number of straight lines is less than a preset threshold, these targets are further screened out. This step can screen out some non-mobile phone targets with more curves and fewer straight lines;

[0020] After obtaining the final personnel target and mobile phone target information, the off-post determination and mobile phone playing determination are performed based on their mutual position information. Off-post determination: If the personnel target is not detected at the workstation, it is determined as off-post; mobile phone playing determination: If the location center of the mobile phone target is included in the personnel target, it is determined as playing with the mobile phone.

[0021] The technical solution of the present invention has the following beneficial effects: The technical solution of the present invention uses target detection technology to detect human body and mobile phone target information, and determines the behavior of leaving the post and playing with mobile phones based on their mutual position relationship, which assists management personnel in supervision. And the recognition and detection results are uploaded to the background management server for processing and storage, so that management personnel can query through the web page:

[0022] 1. Accuracy: After detecting the target using Yolov5, it is screened by Resnet50 and straight line detection to obtain accurate mobile phone target information;

[0023] 2. Real-time, using multi-threaded method to process video streams and detect in real time whether employees are playing with their mobile phones or leaving their posts;

[0024] 3. Convenience: managers can access web pages through browsers to obtain employees’ real-time and past work status. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0026] Figure 1 This is a schematic diagram of the overall structure of a deep learning-based mobile phone playing and off-duty detection and alarm system proposed by the present invention;

[0027] Figure 2 This is a framework flow chart of a method for detecting and alarming mobile phone playing and leaving the post based on deep learning proposed by the present invention;

[0028] Figure 3 This is a flow chart of a target detection algorithm for a deep learning-based mobile phone playing and off-duty detection and alarm method proposed in the present invention.

[0029] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0032] In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0033] The present invention proposes a mobile phone playing and leaving-the-job detection and alarm system based on deep learning.

[0034] like Figure 1As shown, in one embodiment of the present invention, the deep learning-based mobile phone playing and leaving the post detection alarm system includes monitoring cameras, image storage servers and background management servers. The monitoring cameras are respectively arranged above the office area, and the monitoring cameras are used to monitor the behavior of people in the office area. The monitoring cameras are all data-connected to the image storage servers, and the image storage servers are respectively data-connected to the background management servers. The image storage server is used to detect human bodies and mobile phone targets in the image, and mark their location information, and then judge whether the employees are playing with mobile phones or leaving their posts based on the location information of the human bodies and mobile phones. After detecting the behavior of employees playing with mobile phones or leaving their posts, the results are uploaded to the background management server. The background management server is used to record the behavior and display it in real time, and to notify the administrator of the alarm, so that the administrator can monitor the system operation status and the behavior of employees in the office area in real time through the browser.

[0035] like Figure 2 As shown, the present invention also provides a method for detecting and alarming mobile phone playing and leaving the post based on deep learning, which is implemented by the above system and includes the following steps:

[0036] S100: The camera and image algorithm server are connected to the same monitoring intranet;

[0037] S200: The administrator starts the backend management server, logs in to the backend management website, configures the parameters of the image algorithm server, and sets the video stream address of the surveillance camera;

[0038] S300: The administrator starts the image storage server from the backend management website;

[0039] S400: After the camera transmits the monitoring image to the image storage server, the image storage server inputs the image into the target detection algorithm, detects the human body and mobile phone targets in the image in real time, and marks their location information;

[0040] S500: The image storage server determines whether the employee is playing with the mobile phone or off-duty based on the location information of the human body and the mobile phone. If no human body target is detected at the workstation of an employee for more than a preset time threshold, the employee is determined to be off-duty. If a mobile phone appears in the human body frame marked by the target detection algorithm of an employee for more than a preset time threshold, the employee is determined to be playing with the mobile phone.

[0041] S600: After detecting that the employee is playing with the mobile phone or is away from work, the image storage server uploads the result to the background management server, which records the behavior and displays it in the form of a web page;

[0042] S700: Managers monitor employees’ real-time and past work status through the backend management website.

[0043] Specifically, considering that the algorithm may have a certain probability of misjudgment, a delayed end mechanism is introduced. That is, when the algorithm detects abnormal behavior in a certain frame (playing with mobile phones, leaving the post), the timing starts and the next frame is detected; when no abnormal behavior is detected in a certain frame, the timing is not ended immediately, but when there is no abnormal behavior in N consecutive frames; then, it is determined whether it is a real abnormal behavior based on whether the duration of the abnormal behavior exceeds the preset time threshold. This mechanism is conducive to increasing the tolerance of the algorithm's misjudgment rate and improving the accuracy of detection.

[0044] Specifically, the target detection algorithm includes two parts: human target detection and mobile phone target detection. Both human target detection and mobile phone target detection are based on the Yolov5 network framework, and use monitoring data from different perspectives in multiple scenes to create the original data set for neural network training. At the same time, in order to increase the accuracy of mobile phone target detection, after detecting the mobile phone target, the Resnet50 secondary classification network and the straight line detection method are also used for secondary screening.

[0045] Specifically, Figure 3 As shown, the target detection algorithm detects human bodies and mobile phone targets in the image and marks their location information, including the following steps:

[0046] The camera is connected to the network through the Real-Time Streaming Protocol (RTSP). The video stream processing thread directly reads the video stream from the monitoring network and transmits it to the Yolov5 target detection network after processing. The video stream processing thread consists of a writing thread and a reading thread. The writing thread is responsible for reading the monitoring screen and writing it to the queue, and maintaining the queue size at 1 through the producer-consumer model; the reading thread reads the latest frame from the queue, and transmits it to the Yolov5 target detection network after preprocessing;

[0047] The Yolov5 target detection network obtains the latest frame from the reading thread, extracts image features through the convolutional neural network, and predicts the target position of the person and the mobile phone in the frame and the corresponding confidence. It also screens according to the confidence threshold, removes the results with low confidence, and obtains the preliminary prediction results;

[0048] After the Yolov5 target detection network detects the mobile phone target, the mobile phone target is intercepted and the Resnet50 secondary classification network is used to perform the first round of screening to filter out some misidentified targets;

[0049] For mobile phone targets that have been filtered by Resnet50, another round of screening is performed. First, Gaussian smoothing filtering is used to eliminate Gaussian noise, then the canny edge detection algorithm is used to obtain its edge features, and finally the Hough line detection algorithm is used to detect the number of straight lines in the edge features. If the number of straight lines is less than a preset threshold, these targets are further screened out. This step can screen out some non-mobile phone targets with more curves and fewer straight lines;

[0050] After obtaining the final personnel target and mobile phone target information, the off-post determination and mobile phone playing determination are performed based on their mutual position information. Off-post determination: If the personnel target is not detected at the workstation, it is determined as off-post; mobile phone playing determination: If the location center of the mobile phone target is included in the personnel target, it is determined as playing with the mobile phone.

[0051] Specifically, the present invention uses target detection technology to detect human body and mobile phone target information, and determines the behavior of leaving the post and playing with mobile phones based on the position relationship between them, so as to assist managers in supervision. And the recognition and detection results are uploaded to the background management server for processing and storage, so that managers can query through the web page:

[0052] 1. Accuracy: After detecting the target using Yolov5, it is screened by Resnet50 and straight line detection to obtain accurate mobile phone target information;

[0053] 2. Real-time, using multi-threaded method to process video streams and detect in real time whether employees are playing with their mobile phones or leaving their posts;

[0054] 3. Convenience: managers can access web pages through browsers to obtain employees’ real-time and past work status.

[0055] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A method for detecting and alarming mobile phone playing and leaving the post based on deep learning, characterized in that: A deep learning-based mobile phone playing and leaving-the-job detection and alarm system is used. The deep learning-based mobile phone playing and leaving-the-job detection and alarm system includes monitoring cameras, image storage servers and background management servers. The monitoring cameras are respectively arranged above the office area. The monitoring cameras are used to monitor the behavior of people in the office area. The monitoring cameras are all connected to the image storage server data. The image storage servers are respectively connected to the background management server data. The image storage server is used to detect human bodies and mobile phone targets in the image and mark their location information. Then, according to the location information of the human body and the mobile phone, it is judged whether the employee is playing with the mobile phone or leaving the post. After the behavior of the employee playing with the mobile phone or leaving the post is detected, the result is uploaded to the background management server. The background management server is used to record the behavior and display it in real time, and notify the administrator of the alarm, so that the administrator can monitor the system operation status and the behavior of employees in the office area in real time through the browser; The deep learning-based method for detecting and alarming mobile phone use and leaving the post comprises the following steps: The camera and image algorithm server are connected to the same monitoring intranet; The administrator starts the backend management server and logs in to the backend management website to configure the parameters of the image algorithm server and set the video stream address of the surveillance camera; The administrator starts the image storage server from the backend management website; After the camera transmits the monitoring image to the image storage server, the image storage server inputs the image into the target detection algorithm to detect the human body and mobile phone targets in the image in real time and mark their location information; The image storage server determines whether an employee is playing with their phone or off-duty based on the location information of the human body and the mobile phone. If no human body is detected at an employee's workstation for more than a preset time threshold, the employee is considered to be off-duty. If a mobile phone appears in the human body frame marked by the target detection algorithm for more than a preset time threshold, the employee is considered to be playing with his phone. After the image storage server detects that an employee is playing with his / her mobile phone or is away from work, it uploads the result to the background management server, which records the behavior and displays it in the form of a web page. Managers monitor employees' real-time and past work status through the backend management website; The target detection algorithm detects human bodies and mobile phone targets in the image and marks their location information, including the following steps: The camera is connected to the network through the Real-Time Streaming Protocol (RTSP). The video stream processing thread directly reads the video stream from the monitoring network and transmits it to the Yolov5 target detection network after processing. The video stream processing thread consists of a writing thread and a reading thread. The writing thread is responsible for reading the monitoring screen and writing it to the queue, and maintaining the queue size at 1 through the producer-consumer mode. The reading thread reads the latest frame from the queue, and transmits it to the Yolov5 target detection network after preprocessing. The Yolov5 target detection network obtains the latest frame from the reading thread, extracts image features through the convolutional neural network, and predicts the target position of the person and the mobile phone in the frame and the corresponding confidence level. It then filters according to the confidence threshold, removes the results with low confidence level, and obtains preliminary prediction results. After the Yolov5 target detection network detects the mobile phone target, the mobile phone target is intercepted and the Resnet50 secondary classification network is used to perform the first round of screening to filter out some misidentified targets; For the mobile phone targets filtered by Resnet50, another round of screening is performed. First, Gaussian smoothing filtering is used to eliminate Gaussian noise, then the canny edge detection algorithm is used to obtain its edge features, and finally the Hough line detection algorithm is used to detect the number of straight lines in the edge features; if the number of straight lines is less than a preset threshold, then these targets are further screened out; this step can screen out some non-mobile phone targets with more curves and fewer straight lines; After obtaining the final personnel target and mobile phone target information, off-post judgment and mobile phone playing judgment are performed based on their relative position information; off-post judgment: if the personnel target is not detected at the workstation, it is judged as off-post; mobile phone playing judgment: if the position center of the mobile phone target is included in the personnel target, it is judged as playing with the mobile phone.

2. The method for detecting and alarming mobile phone playing and leaving the post based on deep learning according to claim 1 is characterized in that: If no human target is detected at a certain employee's workstation for more than a preset time threshold, it is determined that the employee has left the post, including the following steps: When the algorithm detects abnormal behavior in a certain frame, it starts timing and detects the next frames. When no abnormal behavior is detected in a certain frame, the timing does not end immediately. Instead, it ends when there is no abnormal behavior for N consecutive frames. Then, it determines whether it is a real abnormal behavior based on whether the duration of the abnormal behavior exceeds the preset time threshold.