A system and method for detecting abnormal behavior in office areas based on deep learning

Abstract

This invention relates to the field of image detection technology and discloses a deep learning-based system and method for detecting abnormal behavior in office areas. The system includes surveillance cameras, an image storage server, and a back-end management server. The surveillance cameras are installed in the office area to monitor the behavior of people in the office area. The surveillance cameras are all connected to the image storage server, which is connected to the back-end management server. This invention can automatically identify people in the office area and their corresponding behaviors, as well as the video clips corresponding to abnormal behaviors, and upload them to the back-end management server. This greatly reduces the workload of manual inspections, enables quality supervision of service windows, and effectively helps enterprises improve work efficiency.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and in particular to a system and method for detecting abnormal behavior in an office area based on deep learning. Background Art

[0002] Nowadays, efficient office work is not only required by individuals but also by companies. However, many service halls currently have the phenomenon of employees being lazy at work, which not only makes personal office efficiency low, but also causes trouble for the company's production capacity development.

[0003] However, abnormal behavior detection in traditional office environments currently often relies on manpower to judge whether employees have abnormal behavior through real-time images transmitted by cameras. This will cause two problems: first, it is inefficient and consumes a lot of manpower, which greatly increases the company's operating costs; second, it is impossible to record and retain sudden abnormal behaviors in a timely manner. Summary of the invention

[0004] The main purpose of the present invention is to propose a system and method for detecting abnormal behavior in office areas based on deep learning, which aims to detect abnormal behavior of employees in office areas. After the monitoring images are transmitted to the image storage server in real time through multiple cameras, the image storage server inputs the processed images into a convolutional neural network to detect human targets and corresponding behaviors, and determine whether there are abnormal behaviors such as gathering and sleeping. When the above-mentioned abnormal behaviors exist in the office area, the personnel, abnormal behavior types, time periods of abnormal behaviors and their video images will be retained and recorded in time for subsequent management personnel to review. At the same time, the above-mentioned abnormal behaviors will be alarmed to remind the administrator, and it can also be convenient for the administrator to use a browser to access the backend management system to query the behavior status of employees in each office area.

[0005] To achieve the above-mentioned purpose, the present invention proposes an abnormal behavior detection system for office areas based on deep learning, including surveillance cameras, image storage servers and background management servers. The surveillance cameras are respectively arranged in the office area, and the surveillance cameras are used to monitor the behavior of people in the office area. The surveillance cameras are all data-connected to the image storage servers, and the image storage servers are respectively data-connected to the background management servers. The image storage server is used to automatically identify people in the office area and their corresponding behaviors, and upload corresponding video clips and corresponding abnormal screenshots to the background management server when abnormal behaviors occur. The background management server is used to record the abnormal behaviors and their personnel and videos in time after receiving the information sent by the image storage server, and to notify the administrator of the alarm, so that the administrator can monitor the system operation status and the behavior of employees in the office area in real time through the browser.

[0006] The present invention also provides a method for detecting abnormal behavior in an office area based on deep learning, which is performed using the above system and includes the following steps:

[0007] The image server is connected to the monitoring intranet;

[0008] The administrator logs in to the backend management website and configures the camera video stream address and other parameters bound to the image storage server;

[0009] The administrator starts the designated image storage server through the backend management website;

[0010] The image storage server uses a built-in abnormal behavior recognition algorithm to automatically identify people in the office area and their corresponding behaviors, as well as the corresponding video clips when abnormal behaviors occur, and uploads them to the background management server;

[0011] After receiving the information from the image storage server, the background management server will promptly record the abnormal behavior and its personnel and videos, and notify the administrator of the alarm.

[0012] Furthermore, the abnormal behavior recognition algorithm comprises the following steps:

[0013] Step 1: Video stream input is used as the data input source. First, multiple cameras in the monitoring network are connected through the Real-Time Streaming Protocol (RTSP), and multiple processes are used to process multiple video streams in parallel. Each process is responsible for processing one RTSP video stream, reading the latest video frame, and passing it to the lower-level target detection network after preprocessing;

[0014] Step 2: Target detection. The target detection network is based on Yolov5. The Yolov5 target detection network takes images with a resolution between 416 and 640 as input, extracts image features through a convolutional neural network, and predicts the position of people in each image and the confidence of the corresponding behavior. Yolov5 is used to detect whether the target object in the image is sleeping. After screening by the confidence threshold, the people who meet the confidence threshold are marked as "sleeping people" and "other people".

[0015] Step 3: Classification. If the target detection network in step 2 detects the "sleeping" behavior, the target frame will be sent to the ResNet50 classification network, which is a binary classification network for secondary judgment of "sleeping or not sleeping". Finally, a binary target will be output, that is, "sleeping" and "not sleeping" will be judged.

[0016] Step 4: Optical flow method. If the classification network in step 3 detects the "sleeping" behavior, the optical flow method will be used to detect whether there is motion on the target frame. The optical flow method can calculate the average optical flow of each pixel and determine whether the optical flow is greater than the set threshold. If it is greater than the threshold, it is determined to be in motion, that is, "non-sleeping" behavior. Otherwise, it is determined to be in a stationary state, that is, "sleeping" behavior.

[0017] Step 5: If a target is determined to be in the "non-sleeping" behavior state in the above steps 2, 3, and 4, the target is classified into the "other people" set;

[0018] Step 6: Detection of "gathering" behavior. This step takes all the "other people" target frames in step 5 as input. First, set the "gathering" distance threshold, retrieve all the people in the picture whose distance is less than the threshold, and then calculate the distance matrix of the upper midpoints of all the people frames to generate an undirected graph. Then calculate the maximum connected subgraph set of the generated undirected graph, and finally take out all the maximum connected subgraphs with a length greater than 1, which are the target frame sets of the gathering people. If the people thresholds are all greater than the set "gathering" distance threshold, it means that there is no "gathering" behavior, that is, it is a normal working behavior;

[0019] Step 7: Backstage management, finally sending the predicted location, behavior and abnormal behavior video segments of the personnel to the backstage management server.

[0020] The technical solution of the present invention has the following beneficial effects: the technical solution of the present invention uses multi-channel video to conduct video coverage monitoring of the office area, realizes the monitoring auxiliary capabilities of daily video inspections, problem tracing, and problem evidence collection, and uses video analysis technology to perform intelligent analysis of personnel behavior, so as to timely discover abnormal conditions of personnel in the office area and service windows and issue alarms, greatly reducing the workload of manual inspections, realizing service quality supervision of service windows, and effectively helping enterprises improve work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0022] Figure 1 This is a schematic diagram of the overall framework structure of an abnormal behavior detection system for office areas based on deep learning proposed by the present invention;

[0023] Figure 2This is a flowchart of the steps of a method for detecting abnormal behavior in office areas based on deep learning proposed by the present invention.

[0024] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0027] In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0028] The present invention proposes a system and method for detecting abnormal behavior in an office area based on deep learning.

[0029] like Figure 1 As shown, in one embodiment of the present invention, the office area abnormal behavior detection system based on deep learning includes a monitoring camera 101, an image storage server 102 and a background management server 103. The monitoring cameras are respectively arranged in the office area, and the monitoring cameras are used to monitor the behavior of people in the office area. The monitoring cameras are all connected to the image storage server data, and the image storage servers are respectively connected to the background management server data. The image storage server is used to automatically identify people in the office area and their corresponding behavior, and upload the corresponding video clips and corresponding abnormal screenshots to the background management server when abnormal behavior occurs. The background management server is used to record the abnormal behavior and its personnel and videos in time after receiving the information sent by the image storage server, and to notify the administrator of the alarm, so that the administrator can monitor the system operation status and the behavior of employees in the office area in real time through the browser.

[0030] like Figure 2As shown, the present invention provides a method for detecting abnormal behavior in an office area based on deep learning, which is performed using the above system and includes the following steps:

[0031] S100: The image server is connected to the monitoring intranet;

[0032] S200: The administrator logs in to the backend management website and configures parameters such as the camera video stream address bound to the image storage server;

[0033] S300: The administrator starts the designated image storage server through the backend management website;

[0034] S400: The image storage server automatically identifies people in the office area and their corresponding behaviors, as well as corresponding video clips when abnormal behaviors occur, through a built-in abnormal behavior recognition algorithm, and uploads them to the background management server;

[0035] S500: After receiving the information sent by the image storage server, the background management server promptly records the abnormal behavior and its personnel and video, and notifies the administrator of the alarm.

[0036] The core algorithms of this system are target detection algorithm and classification algorithm. Target detection is based on the Yolov5 network framework, and the classification algorithm is based on the ResNet50 network framework. The overall process is mainly divided into two stages: model training stage and model deployment stage. The two stages complement each other and are indispensable.

[0037] 1. Model Training

[0038] The algorithm for abnormal behavior recognition is mainly based on self-made multi-scene and multi-view data, supplemented by existing open source data sets. Self-made data sets are mainly obtained through two ways. The first is to use cameras installed in the laboratory to regularly shoot videos at fixed time intervals from a bird's-eye view; the second is the daily office scenes in the government hall. The videos and pictures collected every day are summarized, the video is segmented and framed to obtain pictures, invalid frames, blurred frames and other bad frames are deleted, and each picture is manually annotated to generate YOLO format labels. Finally, the data set is summarized and handed over to the algorithm model for training.

[0039] 2. Model Deployment

[0040] Specifically, the abnormal behavior recognition algorithm includes the following steps:

[0041] Step 1: Video stream input is used as the data input source. First, multiple cameras in the monitoring network are connected through the Real-Time Streaming Protocol (RTSP), and multiple processes are used to process multiple video streams in parallel. Each process is responsible for processing one RTSP video stream, reading the latest video frame, and passing it to the lower-level target detection network after preprocessing;

[0042] Step 2: Target detection. The target detection network is based on Yolov5 as a model. The Yolov5 target detection network takes images with a resolution between 416 and 640 as input, extracts image features through a convolutional neural network, and predicts the position of people in each image and the confidence of the corresponding behavior. In order to speed up the processing speed under the parallel input of multiple video frames, batch processing is used to simultaneously predict multiple input images. Yolov5 is used to detect whether the target object in the image is sleeping. After screening by the confidence threshold, the people who meet the confidence threshold are marked as "sleeping people" and "other people".

[0043] Step 3: Classification. If the target detection network in step 2 detects the "sleeping" behavior, the target frame will be sent to the ResNet50 classification network, which is a binary classification network used to make a secondary judgment on "sleeping or not". Finally, a binary target will be output, i.e., "sleeping" or "not sleeping". The purpose of this step is to further strengthen the confidence of the "sleeping" target, which can greatly reduce the occurrence of false detection.

[0044] Step 4: Optical flow method. If the classification network in step 3 detects the "sleeping" behavior, the optical flow method will be used to detect whether there is motion on the target frame. The optical flow method can calculate the average optical flow of each pixel and determine whether the optical flow is greater than the set threshold. If it is greater than the threshold, it is determined to be in motion, that is, "non-sleeping" behavior. Otherwise, it is determined to be in a static state, that is, "sleeping" behavior. The purpose of this step is to further ensure whether the "sleeping" behavior determined in steps 2 and 3 is a real sleeping state, to ensure that there will be no missed judgments or wrong judgments.

[0045] Step 5: If a target is determined to be in the "non-sleeping" behavior state in the above steps 2, 3, and 4, the target is classified into the "other people" set;

[0046] Step 6: Detection of "gathering" behavior. This step takes all the "other people" target frames in step 5 as input. First, set the "gathering" distance threshold, retrieve all the people in the picture whose distance is less than the threshold, and then calculate the distance matrix of the upper midpoints of all the people frames to generate an undirected graph. Then calculate the maximum connected subgraph set of the generated undirected graph, and finally take out all the maximum connected subgraphs with a length greater than 1, which are the target frame sets of the gathering people. If the people thresholds are all greater than the set "gathering" distance threshold, it means that there is no "gathering" behavior, that is, it is a normal working behavior;

[0047] Step 7: Backstage management, finally sending the predicted location, behavior and abnormal behavior video segments of the personnel to the backstage management server.

[0048] Specifically, the technical solution of the present invention uses human target detection and behavior detection to monitor the working status of people in the office area, and uploads the recognition and detection results to the background management server for processing and storage, and finally displays the working status of the office area to the administrator through the browser:

[0049] 1. The system is simpler, without the need to install specific AI cameras, and is compatible with most existing monitoring systems, effectively reducing costs;

[0050] 2. Smarter, real-time detection of the crowd status in all office areas, real-time and accurate recording of employees and whether they have abnormal behavior;

[0051] 3. More convenient, data is stored in the cloud, and administrators can view and manage the system operation status anytime and anywhere through the browser.

[0052] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A method for detecting abnormal behavior in office areas based on deep learning, characterized in that: The abnormal behavior detection system for office areas based on deep learning is used. The abnormal behavior detection system for office areas based on deep learning includes: Monitoring cameras, image storage servers and background management servers, wherein the monitoring cameras are respectively arranged in the office area, and the monitoring cameras are used to monitor the behavior of people in the office area, and the monitoring cameras are all connected to the image storage servers by data, and the image storage servers are respectively connected to the background management servers by data, and the image storage servers are used to automatically identify people in the office area and their corresponding behavior, and upload corresponding video clips and corresponding abnormal screenshots to the background management server when abnormal behavior occurs, and the background management server is used to record abnormal behavior and its personnel and videos in time after receiving the information sent by the image storage server, and to notify the administrator of the alarm, so that the administrator can monitor the system operation status and the behavior of employees in the office area in real time through the browser; The deep learning-based abnormal behavior detection method for office areas includes the following steps: The image server is connected to the monitoring intranet; The administrator logs in to the backend management website and configures the camera video stream address and other parameters bound to the image storage server; The administrator starts the designated image storage server through the backend management website; The image storage server uses a built-in abnormal behavior recognition algorithm to automatically identify people in the office area and their corresponding behaviors, as well as the corresponding video clips when abnormal behaviors occur, and uploads them to the background management server; After receiving the information from the image storage server, the backend management server will promptly record the abnormal behavior, personnel, and video, and notify the administrator of the alarm; The abnormal behavior recognition algorithm comprises the following steps: Step 1: Video stream input is used as the data input source. First, multiple cameras in the monitoring network are connected through the Real-Time Streaming Protocol (RTSP), and multiple processes are used to process multiple video streams in parallel. Each process is responsible for processing one RTSP video stream, reading the latest video frame, and passing it to the lower-level target detection network after preprocessing; Step 2: Target detection. The target detection network is based on Yolov5 as a model. The Yolov5 target detection network takes images with a resolution between 416 and 640 as input, extracts image features through a convolutional neural network, and predicts the position of people in each image and the confidence of the corresponding behavior. Each detected person position is marked as a target box. Yolov5 is used to detect whether the target object in the image is sleeping. After screening by the confidence threshold, the people who meet the confidence threshold are marked as "sleeping people" and "other people". Step 3: Classification. If the target detection network in step 2 detects the "sleeping" behavior, the target frame will be sent to the ResNet50 classification network, which is a binary classification network for secondary judgment of "sleeping or not". Finally, a binary target will be output, that is, "sleeping" and "not sleeping" will be judged; Step 4: Optical flow method. If the classification network in step 3 detects the "sleeping" behavior, the optical flow method will be used to detect whether there is motion on the target frame. The optical flow method can calculate the average optical flow of each pixel and determine whether the optical flow is greater than the set threshold. If it is greater than the threshold, it is determined to be in motion, that is, "non-sleeping" behavior. Otherwise, it is determined to be in a stationary state, that is, "sleeping" behavior. Step 5: If a target is determined to be in the "non-sleeping" behavior state in the above steps 2, 3, and 4, the target is classified into the "other people" set; Step 6: Detection of "gathering" behavior. This step takes all the "other people" target frames in step 5 as input. First, set the "gathering" distance threshold, retrieve all the people in the picture whose distance is less than the threshold, and then calculate the distance matrix of the upper midpoints of all the people frames to generate an undirected graph. Then calculate the maximum connected subgraph set of the generated undirected graph, and finally take out all the maximum connected subgraphs with a length greater than 1, which are the target frame sets of the gathering people. If the people thresholds are all greater than the set "gathering" distance threshold, it means that there is no "gathering" behavior, that is, it is a normal working behavior; Step 7: Backstage management, finally sending the predicted location, behavior and abnormal behavior video segments of the personnel to the backstage management server.

Images