Video monitoring method for indoor working status

Abstract

The present invention discloses a video monitoring method for indoor working status, which belongs to the field of video AI processing technology. It includes the following steps: S101: Performing human body detection on a single-frame video through the Faster RCNN algorithm to obtain the target area of ​​the human body; S102: According to the detection result of step S101, using the gradient direction histogram feature and the color classification feature to track the human body through the TLD algorithm to obtain the position of the target area in the time domain; S103: Judging the motion state of the human body according to the detection result of step S101 and the tracking result of step S102; Correcting the tracking result through the detection result every T frames; S104: Judging the working state of the personnel according to the motion state of step S103. According to the characteristics of multiple joints of the human body and the characteristics of the diversity of clothing colors, personnel tracking is performed to obtain the motion state of the human body, and then the working state of the personnel is judged, including leaving the post, sleeping and slacking off.

Description

Technical Field

[0001] The present invention belongs to the field of video AI processing technology, and in particular relates to a video monitoring method for indoor working conditions, and more particularly to a video monitoring method for the working conditions of on-duty personnel. Background Art

[0002] With the development of computer deep learning technology, the use of target detection-based technology to detect personnel on-the-job can effectively reduce the reliance on traditional manual inspections or manual monitoring, and improve detection efficiency. However, the existing methods of using computer technology to detect personnel on-the-job mainly train classifiers to identify personnel by obtaining characteristic information of the body or head, and determine whether the personnel are on-the-job. However, when monitoring personnel, in addition to monitoring whether the personnel are on-the-job, the existing technology also has the need to monitor whether the personnel are passive (such as sleeping). Summary of the invention

[0003] In order to solve the above problems, the embodiment of the present invention provides a video monitoring method for indoor working status, which tracks people according to the characteristics of multiple joints of the human body and the diversity of clothing colors, obtains the movement status of the human body, and then judges the working status of the people, including leaving the post, passive slacking, etc. The technical solution is as follows:

[0004] On the one hand, an embodiment of the present invention provides a video monitoring method for indoor working status, comprising the following steps:

[0005] S101: Perform human body detection on a single-frame video using the Faster RCNN algorithm to obtain the target area of ​​the human body.

[0006] S102: According to the detection result of step S101, the human body is tracked by using the gradient direction histogram feature and the color classification feature through the TLD algorithm to obtain the position of the target area in the time domain.

[0007] S103: judging the motion state of the human body according to the detection result of step S101 and the tracking result of step S102; and correcting the tracking result according to the detection result every T frames.

[0008] S104: Determine the working status of the person according to the motion status in step S103.

[0009] Among them, the training samples of the Faster RCNN algorithm include human body samples in standing, sitting, lying and lying postures, and the target types of its network structure include human body, chair, table, monitor and sofa.

[0010] Wherein, step S102 specifically includes:

[0011] S201: Acquire the first frame image and the detection result of step S101.

[0012] S202: Initialize the sample library for classification learning.

[0013] S203: Acquire the next frame of image.

[0014] S204: The TLD algorithm obtains trajectory parameters according to the gradient direction histogram features.

[0015] S205: Obtain classification results according to the color classification features.

[0016] S206: Outputting corresponding results as human body tracking results according to the overlapping ratio of the classification results and the trajectory parameters.

[0017] Specifically, step S202 includes: determining information of the target bounding box in the image according to the first frame image and the target area of ​​the human body obtained in step S101, and then generating N samples with an IOU greater than K3 and a size the same as the target size around the target box, performing translation, rotation and scaling operations on the samples respectively, each sample generates M sub-samples, and the N*M sub-samples are combined into a positive sample; correspondingly, a sub-sample with an IOU not greater than K4 is generated as a background sample.

[0018] Specifically, step S204 includes: generating uniformly distributed array points in the target area of ​​the current frame, extracting gradient directional histogram features, calculating the position of the array points in the next frame by matching the gradient directional histogram features, and then using the calculated array points of the next frame to reversely calculate the array points of the current frame, calculating the FB error, removing the array points whose FB errors are greater than the first threshold, and using the remaining array points as tracking points; after obtaining the corresponding tracking points of the adjacent frames, the position of the target area of ​​the next frame is calculated.

[0019] Specifically, step S205 includes: acquiring the video image and the target area of ​​the human body obtained in step S101, taking the location of the target area of ​​the current frame and its adjacent area as the suspected target area of ​​the next frame, scanning the image with scanning windows of different sizes, and processing the scanning results in turn by a color feature classifier, a random fern classifier and a nearest neighbor classifier to obtain the location of the target area of ​​the next frame.

[0020] Specifically, step S206 includes: calculating the overlap rate of the classification result and the trajectory parameter; if the overlap rate is greater than or equal to the second threshold, taking the average value of the trajectory parameter and the classification result as the tracking result; if the overlap rate is less than the second threshold, calculating the confidence of the trajectory parameter and the classification result; if the confidence of the classification result is greater than the confidence of the trajectory parameter, taking the classification result as the tracking result; if the confidence of the trajectory parameter is greater than the confidence of the classification result, taking the trajectory parameter as the tracking result.

[0021] Specifically, the correction process is: calculate the IOU of the detection result and the tracking result, and calculate the correlation with the IOU of the detection result and the tracking result through the Hungarian allocation algorithm; if the detection result and the tracking result are correlated with each other and the IOU of the detection result and the tracking result is greater than K1, the tracking result is considered correct; if the detection result and the tracking result are correlated with each other and the IOU of the detection result and the tracking result is greater than K2 and less than K1, then take the average of the detection result and the tracking result as the tracking result; if the detection result and the tracking result are correlated with each other and the IOU of the detection result and the tracking result is less than K2, then use the detection result to replace the tracking result; if the detection result is not correlated with all the tracking results, it is considered that a new target has appeared and the new target is tracked; if the tracking result is not correlated with all the detection results, the tracking result is considered correct.

[0022] The problem to be solved by the Hungarian allocation algorithm is described as follows: N people are assigned N tasks, one person can only be assigned one task, and one task can only be assigned to one person. Assigning a task to one person requires payment. How to allocate tasks to ensure that the total amount of payment is minimized? For this solution, N detection results and N tracking results are paired, and the Euclidean distance between the center of each pair of detection result area and the center of the tracking result area is calculated to ensure that the sum of the N Euclidean distances is minimized.

[0023] Therefore, when the number of detection results (N) and tracking results (M) is not equal, that is, N≠M, let K=min(N,M). Based on the Hungarian allocation algorithm, K pairs of correlated detection results and tracking results can be obtained. If N>K, there are (NK) detection results that are not correlated with all tracking results; otherwise (that is, M>K), there are (MK) tracking results that are not correlated with all detection results.

[0024] Among them, in step S103, the movement state of the human body is judged by the displacement of the center of mass of the target area between frames, and the movement state includes movement and stillness; step S104 includes: when the target is not found in the image and the number of frames in which the target disappears is greater than the threshold Z1, it is judged that the person is away from work; when the number of frames in which the trajectory of the person is continuously still is greater than the threshold Z2, it is judged that the person is slacking off; when the displacement is greater than or equal to the threshold Y1, it is judged that the person is working normally; when the displacement is less than the threshold Y1, the displacement of the local area of ​​the human body is obtained, if the displacement of the local area of ​​the human body is greater than or equal to the threshold Y2, it is judged that the person is working normally; if it is less than the threshold Y2, it is judged that the person is slacking off.

[0025] On the other hand, an embodiment of the present invention further provides a video monitoring device for indoor working status, the device comprising:

[0026] The human body detection module is used to perform human body detection on a single-frame video using the Faster RCNN algorithm to obtain the target area of ​​the human body.

[0027] The human body tracking module is used to track the human body based on the detection results of the human body detection module by using the gradient direction histogram features and color classification features through the TLD algorithm to obtain the position of the target area in the time domain.

[0028] The motion state judgment module is used to judge the motion state of the human body according to the detection result of the human body detection module and the tracking result of the human body tracking module; and to correct the tracking result according to the detection result every T frames.

[0029] The working state judging module is used to judge the working state of the personnel according to the motion state of the motion state judging module.

[0030] The beneficial effects of the technical solution provided by the embodiment of the present invention are as follows: the embodiment of the present invention provides a video monitoring method for indoor working status, which tracks personnel according to the characteristics of multiple joints of the human body and the characteristics of the diversity of clothing colors, obtains the movement state of the human body, and then judges the working status of the personnel, including leaving the post, passive work, etc. Specifically, according to the characteristics of multiple joints of the human body and the characteristics of the diversity of clothing colors, the human body is tracked according to the gradient direction histogram features and color classification features. Among them, the color of the human body's clothing is greatly different from the background and is less affected by noise, which ensures the stability of the tracking algorithm. In addition, the gradient direction histogram features are used to track the human body, and the amount of calculation is small. Specifically, the tracking algorithm of this patent has good applicability to occlusion, size changes, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of a video monitoring method for indoor working status in an embodiment of the present invention;

[0032] Figure 2 is a detailed flow chart of step S102 in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings.

[0034] Example 1

[0035] See also Figure 1 Embodiment 1 provides a video monitoring method for indoor working status, comprising the following steps:

[0036] S101: Perform human body detection on a single-frame video using the Faster RCNN algorithm to obtain the target area of ​​the human body. The training samples of the Faster RCNN algorithm include human body samples in standing posture, sitting posture, prone posture and lying posture, and the target types of its network structure include human body, chair, table, monitor and sofa, etc. Of course, other human body postures and other targets can be trained according to needs. In order to ensure the accuracy of detection, the number of training samples is preferably greater than 20,000. Normally, the detection rate of the Faster RCNN algorithm is greater than 90%, which is fully sufficient for the purpose of this patent.

[0037] S102: According to the detection result of step S101, the human body is tracked by using the gradient direction histogram feature and the color classification feature through the TLD algorithm to obtain the position of the target area in the time domain. Among them, the human body tracking by using the gradient direction histogram feature through the TLD algorithm is a more conventional method. The gradient direction histogram feature can better characterize the characteristics of the human body. At the same time, the operation amount is small when the gradient direction histogram feature is used for calculation.

[0038] S103: judging the motion state of the human body according to the detection result of step S101 and the tracking result of step S102, which is a conventional technology; the tracking result is corrected by the detection result every T frames to ensure the accuracy of tracking, wherein T is set as needed, such as 5-20.

[0039] S104: judging the working status of the personnel according to the motion status in step S103, wherein the working status includes leaving the post, sleeping, passive work and normal office, etc., and judging is performed by combining the motion status with the judgment strategy.

[0040] Among them, see Figure 2 , step S102 specifically includes:

[0041] S201: Acquire the first frame image and the detection result of step S101.

[0042] S202: Initialize the sample library for classification learning.

[0043] S203: Acquire the next frame of image.

[0044] S204: The TLD algorithm obtains trajectory parameters according to the gradient direction histogram features.

[0045] S205: Obtain classification results according to the color classification features.

[0046] S206: Outputting corresponding results as human body tracking results according to the overlapping ratio of the classification results and the trajectory parameters.

[0047] Specifically, step S202 includes: determining the information of the target bounding box in the image according to the first frame image and the target area of ​​the human body obtained in step S101, and then generating N samples with an IOU greater than K3 and the same size as the target size around the target box, performing translation, rotation and scaling operations on the samples respectively, and each sample generates M sub-samples, and the N*M sub-samples are composed of positive samples; correspondingly, generating sub-samples with an IOU not greater than K4 as background samples. Among them, N, M, K3 and K4 are set as needed, and the positive samples and background samples are conventionally defined in the TLD algorithm. For example, K3 is 0.85 and K4 is 0.25.

[0048] Specifically, step S204 includes: generating uniformly distributed array points in the target area of ​​the current frame, extracting gradient direction histogram features, calculating the position of the array points in the next frame by matching the gradient direction histogram features, and then using the calculated array points of the next frame to reversely calculate the array points of the current frame, calculating the FB error, removing the array points whose FB error is greater than a first threshold (set as needed), and using the remaining array points as tracking points; after obtaining the corresponding tracking points of the adjacent frames, calculating the position of the target area of ​​the next frame.

[0049] Specifically, step S205 includes: obtaining the target area of ​​the human body obtained in the video image and step S101, taking the location of the target area of ​​the current frame and its adjacent area as the suspected target area of ​​the next frame, scanning the image with scanning windows of different sizes (such as 0.5-2.0 times the target size), and processing the scanning results in turn by color feature classifiers, random fern classifiers and nearest neighbor classifiers to obtain the location of the target area of ​​the next frame. The color feature classifier, random fern classifier and nearest neighbor classifier are all conventional classifiers. Among them, the color feature classifier is used to compare the color feature similarity between samples, extract samples whose similarity to the target sample is greater than a threshold, and a large number of samples can be eliminated. The random fern classifier is used to analyze the probability distribution histogram of the target sample and the background sample, and classify the samples by probability. The nearest neighbor classifier is used to classify the samples according to the relative similarity of the samples.

[0050] Specifically, step S206 includes: calculating the overlap rate of the classification result and the trajectory parameter (set as needed, specifically 0.6-0.8), if the overlap rate is greater than or equal to the second threshold, taking the average of the trajectory parameter and the classification result as the tracking result. If the overlap rate is less than the second threshold, calculating the confidence of the trajectory parameter and the classification result; if the confidence of the classification result is greater than the confidence of the trajectory parameter, taking the classification result as the tracking result; if the confidence of the trajectory parameter is greater than the confidence of the classification result, taking the trajectory parameter as the tracking result. This process combines the advantages of classification and tracking, and improves the accuracy of tracking.

[0051] Specifically, the correction process is: calculate the IOU of the detection result and the tracking result, and calculate the correlation by the Hungarian allocation algorithm with the IOU of the detection result and the tracking result. If the detection result and the tracking result are correlated with each other and the IOU of the detection result and the tracking result is greater than K1 (set as needed, specifically 0.6-0.8), the tracking result is considered correct. If the detection result and the tracking result are correlated with each other and the IOU of the detection result and the tracking result is greater than K2 (set as needed, specifically 0.2-0.3) and less than K1, the average value of the detection result and the tracking result is taken as the tracking result. If the detection result and the tracking result are correlated with each other and the IOU of the detection result and the tracking result is less than K2, the tracking result is replaced by the detection result. If the detection result is not correlated with all the tracking results, it is considered that a new target has appeared and the new target is tracked. If the tracking result is not correlated with all the detection results, the tracking result is considered correct.

[0052] Among them, in step S103, the motion state of the human body is judged by the displacement of the center of mass of the target area between frames (specifically, the Euler distance), and the motion state includes motion and stillness. Step S104 includes: when the target is not found in the image and the number of frames in which the target disappears is greater than the threshold Z1, it is judged that the person is off duty. When the number of frames in which the trajectory of the person is continuously still is greater than the threshold Z2, it is judged that the person is slacking off (which can be subdivided into sleeping). When the displacement is greater than or equal to the threshold Y1, it is judged that the person is working normally. When the displacement is less than the threshold Y1, the displacement of the local area of ​​the human body is obtained (obtained by combining step S102 with other methods). If the displacement of the local area of ​​the human body is greater than or equal to the threshold Y2, it is judged that the person is working normally; if it is less than the threshold Y2, it is judged that the person is slacking off. Thresholds Z1, Z2, Y1 and Y2 can be set as needed, such as setting Z1 to 15,000 frames.

[0053] The method of this patent is tested, and the process is as follows:

[0054] Experiment 1: Download five videos from the Internet, track them using the method of this patent, and obtain the target area after tracking; at the same time point, use manual human segmentation. Video 1 is 253 frames, a single-target scene, the scene is empty, the human body moves a lot, and the mark size is 700*400 (in pixels, the same below); Video 2 is 140 frames, a multi-target scene, the scene is empty, and the mark size is 525*210; Video 3 is 1344 frames, a single-target scene, the scene is complex, and there are a lot of office supplies in the scene, and the mark size is 700*400; Video 4 is 2790 frames, a single-target scene, the scene is empty, the human body moves less, and the mark size is 350*250; Video 5 is 841 frames, a multi-target scene, the scene is empty, new targets are added in the middle, some targets move a lot, and a small number of human bodies basically do not move, and the mark size is 150*100. Ten frames are uniformly taken every other video to calculate the center distance error (in pixels) between automatic marking and manual marking. The results are shown in Table 1:

[0055] Table 1

[0056] Center distance error Video 1 3.4 Video 2 3.2 Video 3 3.9 Video 4 2.7 Video 5 1.8

[0057] It can be seen from Table 1 that the tracking method of this patent can accurately obtain the position of the human body in various scenarios and under long-term monitoring.

[0058] In experiment 2, a video with a total length of 5415 frames was downloaded from the Internet. It was a common multi-station office scene with five targets. The first target left his post, the second target slept on his workstation for about 3 minutes, and the third, fourth, and fifth targets worked normally throughout the whole process. Each frame was processed, and the frames of leaving the post, sleeping, and slacking (excluding sleeping) were manually marked, and the status was manually judged. The recognition success rate was calculated. The results are shown in Table 2:

[0059] Table 2

[0060] sleep sabotage Normal operation Leaving the job Success rate 97.10％ 96.30％ 96.70％ 99.70％

[0061] It can be seen from Table 2 that the recognition success rate of the present invention is relatively high, especially the success rate of recognizing sleeping and leaving the post is relatively high.

[0062] Example 2

[0063] Embodiment 2 provides a video monitoring device in an indoor working state, the device comprising:

[0064] The human body detection module is used to perform human body detection on a single-frame video using the Faster RCNN algorithm to obtain the target area of ​​the human body.

[0065] The human body tracking module is used to track the human body based on the detection results of the human body detection module by using the gradient direction histogram features and color classification features through the TLD algorithm to obtain the position of the target area in the time domain.

[0066] The motion state judgment module is used to judge the motion state of the human body according to the detection result of the human body detection module and the tracking result of the human body tracking module; and to correct the tracking result according to the detection result every T frames.

[0067] The working state judging module is used to judge the working state of the personnel according to the motion state of the motion state judging module.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video monitoring method for indoor working conditions, characterized in that: The steps include: S101: Perform human body detection on a single-frame video using the Faster RCNN algorithm to obtain the target area of ​​the human body; S102: According to the detection result of step S101, the human body is tracked by using the gradient direction histogram feature and the color classification feature through the TLD algorithm to obtain the position of the target area in the time domain; S103: judging the motion state of the human body according to the detection result of step S101 and the tracking result of step S102; The tracking results are corrected every T frames based on the detection results; S104: judging the working state of the person according to the motion state in step S103; Step S102 specifically includes: S201: Acquire the first frame image and the detection result of step S101; S202: Initialize the sample library for classification learning; S203: Acquire the next frame of image; S204: The TLD algorithm obtains trajectory parameters according to the gradient direction histogram features; S205: Obtaining classification results according to the color classification features; S206: Outputting corresponding results as human body tracking results according to the classification results and the overlap rate of the trajectory parameters; Step S202 includes: According to the first frame image and the target area of ​​the human body obtained in step S101, the information of the target bounding box in the image is determined, and then N samples with an IOU greater than K3 and the same size as the target size are generated around the target box. The samples are translated, rotated and scaled respectively, and each sample generates M sub-samples. The N*M sub-samples are combined into a positive sample; correspondingly, sub-samples with an IOU not greater than K4 are generated as background samples; Step S204 includes: Generate uniformly distributed array points in the target area of ​​the current frame, extract gradient direction histogram features, calculate the position of the array points in the next frame by matching the gradient direction histogram features, and then use the calculated array points of the next frame to reversely calculate the array points of the current frame, calculate the FB error, remove the array points whose FB error is greater than the first threshold, and use the remaining array points as tracking points; after obtaining the corresponding tracking points of the adjacent frames, calculate the position of the target area of ​​the next frame; Step S205 includes: Obtain the video image and the target area of ​​the human body obtained in step S101, take the location of the target area of ​​the current frame and its adjacent area as the suspected target area of ​​the next frame, scan the image with scanning windows of different sizes, and process the scanning results by a color feature classifier, a random fern classifier, and a nearest neighbor classifier in turn to obtain the location of the target area of ​​the next frame; Step S206 includes: The overlap rate of the classification result and the trajectory parameter is calculated. If the overlap rate is greater than or equal to the second threshold, the average value of the trajectory parameter and the classification result is used as the tracking result. If the overlap rate is less than the second threshold, the confidence of the trajectory parameter and the classification result is calculated. If the confidence of the classification result is greater than the confidence of the trajectory parameter, the classification result is used as the tracking result. If the confidence of the trajectory parameter is greater than the confidence of the classification result, the trajectory parameter is used as the tracking result.

2. The video monitoring method for indoor working status according to claim 1, characterized in that: The training samples of the FasterRCNN algorithm include human body samples in standing, sitting, lying and lying postures, and the target types of its network structure include human body, chair, table, display and sofa.

3. The video monitoring method for indoor working status according to claim 1, characterized in that: The correction process is: Calculate the IOU of the detection result and the tracking result, and use the Hungarian allocation algorithm to calculate the correlation between the IOU of the detection result and the tracking result; If the detection result and the tracking result are correlated and the IOU between the detection result and the tracking result is greater than K1, the tracking result is considered correct; If the detection result and the tracking result are correlated and their IOU is greater than K2 and less than K1, the average of the detection result and the tracking result is taken as the tracking result; If the detection result and the tracking result are correlated and the IOU between the detection result and the tracking result is less than K2, the detection result is used to replace the tracking result; If the detection result is not correlated with all the tracking results, it is considered that a new target has appeared and the new target is tracked; If the tracking result is not correlated with any detection results, the tracking result is considered correct.

4. The video monitoring method for indoor working status according to claim 1, characterized in that: In step S103, the motion state of the human body is determined by the displacement of the centroid of the target area between frames, and the motion state includes motion and stillness; Step S104 includes: When no target is found in the image and the number of frames where the target disappears is greater than the threshold Z1, it is judged that the person is away from his post; When the number of frames in which the person's trajectory is continuously stationary is greater than the threshold Z2, the person is judged to be slacking off; When the displacement is greater than or equal to the threshold value Y1, it is judged that the personnel are working normally; When the displacement is less than the threshold value Y1, the displacement of the local area of ​​the human body is obtained. If the displacement of the local area of ​​the human body is greater than or equal to the threshold value Y2, it is judged that the person is working normally; if it is less than the threshold value Y2, it is judged that the person is slacking off.

5. The video monitoring method for indoor working status according to claim 1, characterized in that: Including video monitoring device for indoor working status; The human body detection module is used to detect human bodies in single-frame videos using the Faster RCNN algorithm to obtain the target area of ​​the human body. The human body tracking module is used to track the human body based on the detection results of the human body detection module by using the gradient direction histogram features and color classification features through the TLD algorithm to obtain the position of the target area in the time domain; A motion state judgment module, used to judge the motion state of a human body according to the detection result of the human body detection module and the tracking result of the human body tracking module; The tracking results are corrected every T frames based on the detection results; The working state judging module is used to judge the working state of the personnel according to the motion state of the motion state judging module.

Images