Image recognition-based method and system for detecting personnel status in office space

Abstract

The present invention discloses a method and system for detecting the status of personnel in an office space based on image recognition. The method extracts a foreground part from a surveillance video of an object to be monitored acquired by a camera to obtain a plurality of image foreground key frames, and encodes the plurality of image foreground key frames by using a first neural network with a three-dimensional convolution kernel to obtain a first tracking feature map, and encodes each of the plurality of image foreground key frames by using a target detection network and a second convolution neural network with a third convolution kernel to obtain a second tracking feature map, and then corrects and fuses the first tracking feature map and the second tracking feature map to obtain a classification feature map, and the classification feature map is passed through a classifier to obtain a classification result for indicating whether the object to be monitored has abnormal behavior.

Description

Technical Field

[0001] The present application relates to the technical field of office personnel status detection, and more specifically, to an office personnel status detection method and system based on image recognition. Background Art

[0002] For modern enterprises, the management and monitoring of the working status of people in the office has always been a problem. For example, in enterprises dominated by hourly wage system, employees may doze off during working hours, leave their posts without permission, have abnormal postures, work while fatigued, etc. The traditional solution is to supervise manually, such as through inspections by superiors, but this manual solution not only wastes the time of superiors, but also has poor supervision effect.

[0003] Therefore, a solution for detecting the status of personnel in an office is desired. Summary of the invention

[0004] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide an office personnel status detection system, method and electronic device based on image recognition, which extracts the foreground part from the surveillance video of the monitored object collected by the camera to obtain multiple image foreground key frames, encodes the multiple image foreground key frames by using a first neural network with a three-dimensional convolution kernel to obtain a first tracking feature map, and encodes each of the multiple image foreground key frames by using a target detection network and a second convolution neural network with a third convolution kernel to obtain a second tracking feature map, and then corrects and fuses the first tracking feature map and the second tracking feature map to obtain a classification feature map, and the classification feature map is passed through a classifier to obtain a classification result for indicating whether the monitored object has abnormal behavior, which is conducive to improving the supervision effect of personnel in the office.

[0005] According to one aspect of the present application, a system for detecting personnel status in an office space based on image recognition is provided, comprising:

[0006] A monitoring module, used to obtain monitoring videos of the monitored object collected by cameras deployed in the office;

[0007] An image key frame extraction module, used to extract multiple image key frames from the surveillance video;

[0008] A background removal module, used for extracting the foreground part of each image key frame in the plurality of image key frames to obtain a plurality of image foreground key frames;

[0009] A first convolutional encoding module, configured to obtain a first tracking feature map by encoding the plurality of image foreground key frames through a first convolutional neural network using a three-dimensional convolution kernel;

[0010] An area of ​​interest extraction module, used for respectively passing each of the multiple image foreground key frames through an object detection network to obtain multiple face areas of interest;

[0011] A second convolutional encoding module, used for obtaining a second tracking feature map by encoding the plurality of face regions of interest through a second convolutional neural network using a three-dimensional convolution kernel;

[0012] A feature distribution correction module, used for correcting the first tracking feature map and the second tracking feature map to obtain a corrected first tracking feature map and a corrected second tracking feature map respectively;

[0013] a feature fusion module, configured to fuse the corrected first tracking feature map and the corrected second tracking feature map to obtain a classification feature map; and

[0014] The state detection result generating module is used to pass the classification feature graph through a classifier to obtain a classification result, and the classification result is used to indicate whether the monitored object has abnormal behavior.

[0015] On the other hand, the present application provides a method for detecting the status of personnel in an office space based on image recognition, comprising:

[0016] Obtaining surveillance videos of the monitored object collected by cameras deployed in the office;

[0017] Extracting a plurality of image key frames from the surveillance video;

[0018] Extracting a foreground portion of each image key frame in the plurality of image key frames to obtain a plurality of image foreground key frames;

[0019] The plurality of image foreground key frames are passed through a first convolutional neural network using a three-dimensional convolution kernel to obtain a first tracking feature map;

[0020] Passing each of the plurality of image foreground key frames through an object detection network to obtain a plurality of face regions of interest;

[0021] The plurality of face regions of interest are passed through a second convolutional neural network using a three-dimensional convolution kernel to obtain a second tracking feature map;

[0022] Correcting the first tracking characteristic map and the second tracking characteristic map respectively to obtain a corrected first tracking characteristic map and a corrected second tracking characteristic map;

[0023] fusing the corrected first tracking feature map and the corrected second tracking feature map to obtain a classification feature map; and

[0024] The classification feature graph is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the object to be monitored has abnormal behavior.

[0025] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory, in which computer program instructions are stored, and when the computer program instructions are executed by the processor, the processor executes the above-mentioned method for detecting personnel status in an office space based on image recognition.

[0026] According to another aspect of the present application, a computer-readable medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the above-mentioned method for detecting personnel status in an office space based on image recognition.

[0027] Compared with the prior art, the office personnel status detection system and method based on image recognition provided by the present application extract the foreground part from the surveillance video of the monitored object collected by the camera to obtain multiple image foreground key frames, encode the multiple image foreground key frames by using a first neural network with a three-dimensional convolution kernel to obtain a first tracking feature map, encode each of the multiple image foreground key frames by using a target detection network and a second convolution neural network with a third convolution kernel to obtain a second tracking feature map, and then correct and fuse the first tracking feature map and the second tracking feature map to obtain a classification feature map. The classification feature map is passed through a classifier to obtain a classification result for indicating whether the monitored object has abnormal behavior, which is conducive to improving the supervision effect of personnel in the office. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0029] Figure 1 An application scenario diagram of an office personnel status detection method based on image recognition according to an embodiment of the present application is illustrated.

[0030] Figure 2 The figure illustrates a structural block diagram of an office personnel status detection system based on image recognition according to an embodiment of the present application;

[0031] Figure 3 The figure shows a schematic diagram of the architecture of an office personnel status detection system based on image recognition according to an embodiment of the present application;

[0032] Figure 4 A flowchart of a method for detecting personnel status in an office space based on image recognition according to an embodiment of the present application is illustrated;

[0033] Figure 5 A block diagram of an electronic device according to an embodiment of the present application is illustrated. DETAILED DESCRIPTION

[0034] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0035] Application Overview:

[0036] As described above, a solution for detecting the status of personnel in an office is desired.

[0037] In recent years, deep learning and neural networks have been widely used in computer vision, natural language processing, text signal processing and other fields. In addition, deep learning and neural networks have also demonstrated a level close to or even beyond that of humans in image classification, object detection, semantic segmentation, text translation and other fields.

[0038] The development of deep learning and neural networks has provided new ideas and solutions for the status detection of personnel in office places. It should be understood that the problem of personnel status detection in office places is essentially a multi-classification problem, that is, the behavioral characteristics of personnel in the office are extracted, and the behavioral characteristics are input into a multi-label classifier for classification to obtain the behavioral labels of personnel in the office. In the application scenario of this application, behavioral labels include but are not limited to dozing off during working hours, leaving the post without permission during working hours, abnormal posture, fatigue work, etc.

[0039] Specifically, first, a surveillance video of the object to be monitored is collected by a camera deployed in an office space. Considering that many consecutive frames in the entire surveillance video are repeated or similar, information redundancy is caused, which increases the amount of subsequent model calculations. In response to this technical problem, the multiple image frames are extracted from the surveillance video as the multiple image key frames, which effectively removes redundant information of the video sequence and greatly improves time efficiency. For example, in a specific example of the present application, the multiple image frames are extracted from the surveillance video at a predetermined sampling frequency as the multiple image key frames; in another specific example of the present application, the multiple image frames are extracted from the surveillance video as the multiple image key frames by using the difference frame method (using the difference between adjacent image frames as the key frame determination standard).

[0040] Furthermore, in order to remove the interference of background information, a foreground extraction template is used to obtain multiple image foreground key frames from the foreground part of each image key frame in the multiple image key frames. Then, a convolutional neural network model is used as a feature extractor to extract the behavioral features of the monitored object from the multiple image foreground key frames to obtain a first tracking feature map. That is, the multiple image foreground key frames are used to replace the entire monitoring video as the input of the neural network, so as to reduce the data processing amount of the neural network model on the one hand, and on the other hand, avoid the interference information brought into the input end from confusing the feature extraction of the neural network.

[0041] In addition to using the entire monitored object as a unit for feature extraction, in the technical solution of the present application, considering that the state of a person is more reflected in the face area, therefore, in the technical solution of the present application, each of the multiple image foreground key frames is passed through a target detection network to obtain multiple face interest regions, where the target region detection network is a face region detection network. Then, the dynamic change features of the face area of ​​the monitored object are extracted using a neural network model using a three-dimensional convolution kernel as a feature extractor to obtain a second tracking feature map.

[0042] Next, after the first tracking feature graph and the second tracking feature graph are fused to obtain a classification feature graph, the classification feature graph is passed through a classifier to obtain a classification result indicating whether the object to be monitored has abnormal behavior.

[0043] In particular, in the technical solution of the present application, since the first tracking feature map represents the image semantic association features of the image foreground key frame, and the second tracking feature map represents the image semantic association features of the region of interest of the image foreground key frame, this makes the second tracking feature map, as a local image semantic association feature, have a high-dimensional manifold scale difference with the overall image semantic association features of the first tracking feature map, resulting in the possible scale compatibility defects of the fused high-dimensional manifold, affecting the classification effect.

[0044] Therefore, based on the local-global overlapping relationship between the first tracking feature map and the second tracking feature map in the image dimension, the frugal decomposition encouragement factor of the first tracking feature map and the second tracking feature map is calculated as a weighting coefficient, which is expressed as:

[0045]

[0046] and are the eigenvalues ​​of the first tracking feature map and the second tracking feature map, τ1 and τ2 are hyperparameters, ReLU represents the ReLU activation function, and ||·||2 represents the bi-norm of the vector

[0047] The frugal decomposition encouragement factor is used to group the feature map on the image plane and impose a penalty on the overlap of elements within the group to promote the frugal decomposition of the high-dimensional manifold of the feature map by calculating the distance-based union of the symbolic function, that is, decomposing the geometric shape of the high-dimensional manifold into a set of convex polytopes, so that when the first tracking feature map and the second tracking feature map are fused, the scale compatibility of the fused high-dimensional manifold is improved because the first tracking feature map and the second tracking feature map are both represented by a set of basic manifolds. In this way, the accuracy of monitoring the status of personnel in the office is improved.

[0048] Based on this, the present application proposes an office personnel status detection system and method based on image recognition, which extracts the foreground part from the surveillance video of the monitored object captured by the camera to obtain multiple image foreground key frames, encodes the multiple image foreground key frames by using a first neural network with a three-dimensional convolution kernel to obtain a first tracking feature map, and encodes each of the multiple image foreground key frames by using a target detection network and a second convolution neural network with a third convolution kernel to obtain a second tracking feature map, and then corrects and fuses the first tracking feature map and the second tracking feature map to obtain a classification feature map. The classification feature map is passed through a classifier to obtain a classification result for indicating whether the monitored object has abnormal behavior, which is conducive to improving the supervision effect of personnel in the office.

[0049] Figure 1 The figure shows a scene schematic diagram of the method for detecting the status of personnel in an office space based on image recognition according to an embodiment of the present application. Figure 1 As shown, in an application scenario of the present application, first, by deploying in an office location (such as Figure 1 The camera in W) (such as Figure 1 C) Collect the objects to be monitored (such as Figure 1 Then, the monitoring video of the object to be monitored is input to a server (such as a server) that is deployed with an office personnel status detection algorithm based on image recognition. Figure 1 In S), the server is capable of processing the surveillance video of the monitored object through an office personnel status detection algorithm based on image recognition to output a classification result indicating whether the monitored object has abnormal behavior.

[0050] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0051] Exemplary Systems

[0052] Figure 2 The figure illustrates a structural block diagram of an office personnel status detection system based on image recognition according to an embodiment of the present application.

[0053] like Figure 2 As shown, the office personnel status detection system 100 based on image recognition provided by the embodiment of the present application includes: a monitoring module 110, which is used to obtain a monitoring video of the object to be monitored collected by a camera deployed in the office; an image key frame extraction module 120, which is used to extract multiple image key frames from the monitoring video; a background removal module 130, which is used to extract the foreground part of each image key frame in the multiple image key frames to obtain multiple image foreground key frames; a first convolution encoding module 140, which is used to obtain a first tracking feature map by using a first convolution neural network with a three-dimensional convolution kernel; and an area of ​​interest extraction module 150, which is used to separate each image foreground key frame in the multiple image foreground key frames into a plurality of image foreground key frames. A second convolutional coding module 160 is used to obtain a second tracking feature map by using a second convolutional neural network with a three-dimensional convolution kernel for the multiple face regions of interest; a feature distribution correction module 170 is used to correct the first tracking feature map and the second tracking feature map respectively to obtain a corrected first tracking feature map and a corrected second tracking feature map; a feature fusion module 180 is used to fuse the corrected first tracking feature map and the corrected second tracking feature map to obtain a classification feature map; a state detection result generation module 190 is used to pass the classification feature map through a classifier to obtain a classification result, and the classification result is used to indicate whether the monitored object has abnormal behavior.

[0054] Figure 3 FIG. 1 is a schematic diagram of the architecture of an office personnel status detection system based on image recognition according to an embodiment of the present application. Figure 3As shown, in the network architecture, first, multiple image key frames are extracted from the surveillance video of the object to be monitored, and the foreground part of each image key frame in the multiple image key frames is extracted to obtain multiple image foreground key frames; then, the multiple image foreground key frames are passed through a first convolutional neural network using a three-dimensional convolution kernel to obtain a first tracking feature map, each image foreground key frame in the multiple image foreground key frames is passed through a target detection network to obtain multiple face interest regions, and the multiple face interest regions are passed through a second convolutional neural network using a three-dimensional convolution kernel to obtain a second tracking feature map; then, the first tracking feature map and the second tracking feature map are corrected respectively to obtain a corrected first tracking feature map and a corrected second tracking feature map; then, the corrected first tracking feature map and the corrected second tracking feature map are fused to obtain a classification feature map; finally, the classification feature map is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the object to be monitored has abnormal behavior.

[0055] Specifically, the monitoring module 110 is used to obtain the monitoring video of the monitored object collected by the camera deployed in the office. In order to extract the behavioral characteristics of the personnel in the office, in the embodiment of the present application, the monitoring video of the monitored object collected by the camera deployed in the office is used to extract the behavioral characteristics of the monitored object from the monitoring video of the monitored object to determine whether the monitored object has abnormal behavior.

[0056] The image key frame extraction module 120 is used to extract multiple image key frames from the surveillance video. Considering that many consecutive frames in the entire surveillance video are repeated or similar, information redundancy is caused, which increases the amount of subsequent model calculations. In response to this technical problem, the multiple image frames are extracted from the surveillance video as the multiple image key frames, which effectively removes redundant information of the video sequence and greatly improves time efficiency. For example, in a specific example of the present application, the multiple image frames are extracted from the surveillance video at a predetermined sampling frequency as the multiple image key frames; in another specific example of the present application, the multiple image frames are extracted from the surveillance video as the multiple image key frames using the difference frame method (using the difference between adjacent image frames as the key frame determination standard).

[0057] More specifically, the image key frame extraction module 120 includes:

[0058] A preprocessing module, used to grayscale and perform Gaussian blur processing on each frame in the monitored video to obtain multiple preprocessed frames;

[0059] A threshold calculation module is used to calculate the average value and standard deviation of the pixel difference between two adjacent frames, and calculate the pixel threshold, which is 0.75*standard deviation+average value;

[0060] The key frame screening module is used to determine whether the pixel difference between two adjacent frames (such as the average value of the pixel difference, etc.) is greater than the pixel threshold. If it is greater, the current frame is the image key frame. If not, it is determined whether the pixel difference between the next two adjacent frames is greater than the pixel threshold, until all two adjacent frames are compared, and multiple image key frames are obtained.

[0061] In other words, the frame difference method is used to extract image key frames, and the original video sequence is replaced by the key frame sequence, which effectively removes redundant information of the video sequence and greatly improves time efficiency.

[0062] The background removal module 130 and the first convolutional encoding module 140 are used to extract the foreground part of each of the multiple image key frames to obtain multiple image foreground key frames, and the multiple image foreground key frames are subjected to the first convolutional neural network using a three-dimensional convolution kernel to obtain a first tracking feature map. Furthermore, in order to remove the interference of background information, a foreground extraction template is used to extract the foreground part of each of the multiple image key frames to obtain multiple image foreground key frames. Then, the convolutional neural network model is used as a feature extractor to extract the behavioral features of the monitored object from the multiple image foreground key frames to obtain a first tracking feature map. That is, the multiple image foreground key frames are used to replace the entire monitoring video as the input of the neural network, so as to reduce the data processing amount of the neural network model on the one hand, and on the other hand, avoid the interference information brought into the input end from confusing the feature extraction of the neural network.

[0063] The first convolutional coding module 140 is further used to use each layer of the first convolutional neural network to perform three-dimensional convolution processing, pooling processing and nonlinear activation processing based on the three-dimensional convolution kernel on the input data in the forward transfer of the layer to output the first tracking feature map from the last layer of the first convolutional neural network, wherein the input of the first layer of the first convolutional neural network is the multiple image foreground key frames. In this way, the high-dimensional local behavior features of the monitored object, i.e., the first tracking feature map, are extracted from the multiple image foreground key frames by the first convolutional neural network.

[0064] The region of interest extraction module 150 and the second convolutional encoding module 160 are used to pass each of the multiple image foreground key frames through a target detection network to obtain multiple face regions of interest; and pass the multiple face regions of interest through a second convolutional neural network using a three-dimensional convolutional kernel to obtain a second tracking feature map. In addition to using the entire monitored object as a feature extraction unit, in the technical solution of the present application, considering that the state of a person is more reflected in the face area, therefore, in the technical solution of the present application, each of the multiple image foreground key frames is passed through a target detection network to obtain multiple face regions of interest, and here, the target area detection network is a face area detection network. Then, the dynamic change features of the face area of ​​the monitored object are extracted using the neural network model using a three-dimensional convolutional kernel as a feature extractor to obtain a second tracking feature map.

[0065] In some optional examples of the present application, the target detection network is a target detection network based on a non-anchor window, and the target detection network based on a non-anchor window includes but is not limited to YOLOv1, FCOS, CenterNet, CornerNet, ExtremeNet, RepPoints and other networks.

[0066] The second convolutional coding module 160 is further used to use each layer of the second convolutional neural network to perform three-dimensional convolution processing, pooling processing and nonlinear activation processing based on the three-dimensional convolution kernel on the input data in the forward transfer of the layer to output the second tracking feature map from the last layer of the second convolutional neural network, wherein the input of the first layer of the second convolutional neural network is the multiple face interest regions. In this way, the high-dimensional dynamic change features of the face of the monitored object are extracted from the multiple image foreground key frames through the second convolutional neural network, that is, the first tracking feature map.

[0067] It is understandable that after obtaining the first tracking feature map and the second tracking feature map, after fusing the first tracking feature map and the second tracking feature map to obtain a classification feature map, the classification feature map can be passed through a classifier to obtain a classification result indicating whether the monitored object has abnormal behavior. In particular, in the technical solution of the present application, since the first tracking feature map represents the image semantic association features of the image foreground key frame, and the second tracking feature map represents the image semantic association features of the region of interest of the image foreground key frame, this makes the second tracking feature map, as a local image semantic association feature, have a scale difference of the high-dimensional manifold with the overall image semantic association features of the first tracking feature map, resulting in the possibility of scale compatibility defects in the fused high-dimensional manifold, affecting the classification effect. Therefore, in the present application, based on the local-global overlapping relationship of the first tracking feature map and the second tracking feature map in the image dimension, the first tracking feature map and the second tracking feature map are corrected respectively using the frugal decomposition encouragement factor of the first tracking feature map and the second tracking feature map as weighting coefficients, so that when the first tracking feature map after correction and the second tracking feature map after correction are fused, since the first tracking feature map and the second tracking feature map are both represented by a set of basic manifolds, the scale compatibility of the fused high-dimensional manifold is improved. This is conducive to improving the accuracy of monitoring the status of personnel in office spaces.

[0068] Specifically, the feature distribution correction module 170 and the feature fusion module 180 are used to correct the first tracking feature map and the second tracking feature map respectively to obtain a corrected first tracking feature map and a corrected second tracking feature map; and fuse the corrected first tracking feature map and the corrected second tracking feature map to obtain a classification feature map.

[0069] Furthermore, the feature distribution correction module 170 includes:

[0070] a first weight calculation unit, configured to calculate a frugal decomposition encouragement factor of the first tracking feature map as a weighting coefficient as a first weight, wherein the frugal decomposition encouragement factor of the first tracking feature map as a weighting coefficient is related to a weighted sum of a natural exponential function raised to a power of a negative value of an eigenvalue at each position in each feature matrix of the first tracking feature map with a feature matrix as a weighting unit;

[0071] a second weight calculation unit, configured to calculate a frugal decomposition encouragement factor of the second tracking feature map as a weighting coefficient as a second weight, wherein the frugal decomposition encouragement factor of the second tracking feature map as a weighting coefficient is related to a weighted sum of a natural exponential function raised to a power of a negative value of an eigenvalue at each position in each feature matrix of the first tracking feature map with a feature matrix as a weighting unit;

[0072] A first correction unit, configured to weight the feature values ​​of each position in the first tracking feature map with the first weight to obtain the corrected first tracking feature map; and

[0073] The second correction unit is used to weight the feature value of each position in the second tracking feature map with the second weight to obtain the corrected second tracking feature map.

[0074] In one embodiment of the present application, the first weight calculation unit is further used to: calculate the frugal decomposition encouragement factor of the first tracking feature map as a weighting coefficient as the first weight using the following formula, wherein the formula is:

[0075]

[0076] Where w1 is the first weight, is the eigenvalue of the first tracking feature map, τ1 is a hyperparameter, ReLU represents the ReLU activation function, and ||·||2 represents the second norm of the vector.

[0077] In one embodiment of the present application, the second weight calculation unit is further used to: calculate the frugal decomposition encouragement factor of the second tracking feature map as a weighting coefficient as the second weight using the following formula, wherein the formula is:

[0078]

[0079] Where w2 is the second weight, is the eigenvalue of the second tracking feature map, τ2 is a hyperparameter, ReLU represents the ReLU activation function, and ||·||2 represents the second norm of the vector.

[0080] The frugal decomposition encouragement factor is used to group feature maps on the image plane and impose a penalty on the overlap of elements within the group to promote the frugal decomposition of the high-dimensional manifold of the feature map by calculating the distance-based union of symbolized functions, that is, decomposing the geometric shape of the high-dimensional manifold into a set of convex polytopes. In this way, when the first tracking feature map and the second tracking feature map are fused, the scale compatibility of the fused high-dimensional manifold is improved because both the first tracking feature map and the second tracking feature map are represented by a set of basis manifolds.

[0081] The state detection result generating module 190 is used to pass the classification feature graph through a classifier to obtain a classification result, and the classification result is used to indicate whether the monitored object has abnormal behavior.

[0082] In some embodiments of the present application, the classification process of the state detection result generation module 190 includes: using multiple fully connected layers of the classifier to fully connect encode the classification feature map to convert the classification feature map into a classification feature vector; inputting the classification feature vector into a Softmax classification function to obtain the probability values ​​of the classification feature vectors respectively belonging to whether there is abnormal behavior in the monitored object; if the probability value of the existence of abnormal behavior in the monitored object is greater than or equal to the absence of abnormal behavior in the monitored object, then the output classification result is that the monitored object has abnormal behavior; if the probability value of the existence of abnormal behavior in the monitored object is less than the probability value of the absence of abnormal behavior in the monitored object, then the output classification result is that the monitored object does not have abnormal behavior.

[0083] The learning topic intelligent allocation result generation module 119 is further used to:

[0084] The classification feature map is processed by the classifier using the following formula to generate a classification result, wherein the formula is: softmax{(W n , B n ):...:(W1,B1)|Project(F)}, where Project(F) represents projecting the classification feature map into a vector, W1 to W n is the weight matrix of each fully connected layer, B1 to B n Represents the bias matrix of each fully connected layer.

[0085] In summary, the image recognition-based office personnel status detection system of the embodiment of the present application is explained, which extracts the foreground part from the surveillance video of the monitored object captured by the camera to obtain multiple image foreground key frames, encodes the multiple image foreground key frames by using a first neural network with a three-dimensional convolution kernel to obtain a first tracking feature map, and encodes each of the multiple image foreground key frames by using a target detection network and a second convolution neural network with a third convolution kernel to obtain a second tracking feature map, and then corrects and fuses the first tracking feature map and the second tracking feature map to obtain a classification feature map. The classification feature map is passed through a classifier to obtain a classification result for indicating whether the monitored object has abnormal behavior, which is conducive to improving the supervision effect of personnel in the office.

[0086] As described above, the office personnel status detection system 100 based on image recognition according to the embodiment of the present application can be implemented in various terminal devices, such as a server for office personnel status detection based on image recognition, etc. In one example, the office personnel status detection system 100 based on image recognition according to the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the office personnel status detection system 100 based on image recognition can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the office personnel status detection system 100 based on image recognition can also be one of the many hardware modules of the terminal device.

[0087] Alternatively, in another example, the office personnel status detection system 100 based on image recognition and the terminal device may also be separate devices, and the office personnel status detection system 100 based on image recognition may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0088] Exemplary Methods

[0089] Figure 4 FIG. 1 is a flowchart of a method for detecting the status of personnel in an office space based on image recognition according to an embodiment of the present application. Figure 4 As shown, according to an embodiment of the present application, a method for detecting personnel status in an office space based on image recognition includes:

[0090] S101, obtaining a surveillance video of the object to be monitored collected by a camera deployed in an office;

[0091] S102, extracting a plurality of image key frames from the surveillance video;

[0092] S103, extracting the foreground part of each image key frame in the multiple image key frames to obtain multiple image foreground key frames;

[0093] S104, obtaining a first tracking feature map by using a first convolutional neural network with a three-dimensional convolution kernel for the multiple image foreground key frames;

[0094] S105, passing each of the multiple image foreground key frames through an object detection network to obtain multiple face regions of interest;

[0095] S106, obtaining a second tracking feature map by using a second convolutional neural network with a three-dimensional convolution kernel for the multiple face regions of interest;

[0096] S107, respectively correcting the first tracking characteristic map and the second tracking characteristic map to obtain a corrected first tracking characteristic map and a corrected second tracking characteristic map;

[0097] S108, fusing the corrected first tracking feature map and the corrected second tracking feature map to obtain a classification feature map; and

[0098] S109: Pass the classification feature graph through a classifier to obtain a classification result, where the classification result is used to indicate whether the object to be monitored has abnormal behavior.

[0099] In one example, in the above-mentioned method for detecting personnel status in an office space based on image recognition, extracting a plurality of image key frames from the surveillance video includes: extracting the plurality of image key frames from the surveillance video at a predetermined sampling frequency.

[0100] In one example, in the above-mentioned method for detecting personnel status in an office space based on image recognition, extracting a plurality of image key frames from the surveillance video includes: extracting the plurality of image key frames from the surveillance video using a difference frame method.

[0101] In one example, in the above-mentioned office personnel status detection method based on image recognition, the multiple image foreground key frames are obtained by using a first convolutional neural network with a three-dimensional convolution kernel to obtain a first tracking feature map, including: using each layer of the first convolutional neural network to perform three-dimensional convolution processing, pooling processing and non-linear activation processing based on the three-dimensional convolution kernel on the input data in the forward pass of the layer to output the first tracking feature map by the last layer of the first convolutional neural network, wherein the input of the first layer of the first convolutional neural network is the multiple image foreground key frames.

[0102] In one of the examples, in the above-mentioned office personnel status detection method based on image recognition, the target detection network is a target detection network based on anchor-free windows, and the target detection network based on anchor-free windows includes YOLOv1, FCOS, CenterNet, CornerNet, ExtremeNet, and RepPoints.

[0103] In one example, in the above-mentioned method for detecting the status of personnel in an office space based on image recognition, respectively correcting the first tracking feature map and the second tracking feature map to obtain a corrected first tracking feature map and a corrected second tracking feature map, comprises:

[0104] Calculating a frugal decomposition encouragement factor of the first tracking feature map as a weighting coefficient as a first weight, wherein the frugal decomposition encouragement factor of the first tracking feature map as a weighting coefficient is related to a weighted sum of a natural exponential function raised to a power of a negative value of an eigenvalue at each position in each feature matrix of the first tracking feature map with a feature matrix as a weighting unit;

[0105] Calculating a frugal decomposition encouragement factor of the second tracking feature map as a weighting coefficient as a second weight, wherein the frugal decomposition encouragement factor of the second tracking feature map as a weighting coefficient is related to a weighted sum of a natural exponential function raised to a power of a negative value of an eigenvalue at each position in each feature matrix of the first tracking feature map with a feature matrix as a weighting unit;

[0106] weighting the feature values ​​at each position in the first tracking feature map with the first weight to obtain the corrected first tracking feature map; and

[0107] The feature values ​​at each position in the second tracking feature map are weighted by the second weight to obtain the corrected second tracking feature map.

[0108] In one example, in the above-mentioned method for detecting the status of personnel in an office space based on image recognition, the frugal decomposition encouragement factor of the first tracking feature graph is calculated as a weighting coefficient as a first weight using the following formula, wherein the formula is:

[0109]

[0110] Where W1 is the first weight, is the eigenvalue of the first tracking feature map, τ1 is a hyperparameter, ReLU represents the ReLU activation function, and ||·||2 represents the second norm of the vector.

[0111] In one example, in the above-mentioned method for detecting the status of personnel in an office space based on image recognition, the frugal decomposition encouragement factor of the second tracking feature graph is calculated as a weighting coefficient as the second weight using the following formula, wherein the formula is:

[0112]

[0113] Where W2 is the second weight, is the eigenvalue of the second tracking feature map, τ2 is a hyperparameter, ReLU represents the ReLU activation function, and ||·||2 represents the second norm of the vector.

[0114] In one example, in the above-mentioned method for detecting the status of personnel in an office space based on image recognition, the step of passing the classification feature graph through a classifier to obtain a classification result includes:

[0115] The classification feature map is processed by the classifier using the following formula to generate a classification result, wherein the formula is: softmax{(W n , B n ):...:(W1,B1)|Project(F), where Project(F) represents projecting the classification feature map into a vector, W1 to W n is the weight matrix of each fully connected layer, B1 to B n Represents the bias matrix of each fully connected layer.

[0116] Here, those skilled in the art can understand that the specific functions and steps in the above-mentioned method for detecting the status of personnel in an office space based on image recognition have been described in the above reference. Figure 2 to Figure 3 The above description has been introduced in detail in the description of the office personnel status detection system based on image recognition, and therefore, its repeated description will be omitted.

[0117] It is to be understood that some or all of the steps or operations in the above embodiments are merely examples, and the present application embodiments may also perform other operations or variations of various operations. In addition, the various steps may be performed in different orders presented in the above embodiments, and it is possible that not all of the operations in the above embodiments need to be performed.

[0118] Exemplary Electronic Devices

[0119] Below, reference Figure 5 To describe an electronic device according to an embodiment of the present application.

[0120] Figure 5 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0121] like Figure 5 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0122] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0123] The memory 12 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the office personnel status detection method based on image recognition of the various embodiments of the present application described above and / or other desired functions. Various contents such as parameters may also be stored in the computer-readable storage medium.

[0124] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0125] The input device 13 may include, for example, a keyboard, a mouse, etc.

[0126] The output device 14 can output various information to the outside, including classification results or warning prompts, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0127] Of course, to simplify, Figure 5 Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0128] Exemplary computer program products and computer-readable storage media

[0129] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the office personnel status detection method based on image recognition according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0130] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0131] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the office personnel status detection method based on image recognition according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0132] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0133] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0134] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0135] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0136] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0137] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An office personnel status detection system based on image recognition, characterized in that: include: A monitoring module, used to obtain monitoring videos of the monitored object collected by cameras deployed in the office; An image key frame extraction module, used to extract multiple image key frames from the surveillance video; A background removal module, used for extracting the foreground part of each image key frame in the plurality of image key frames to obtain a plurality of image foreground key frames; A first convolutional encoding module, configured to obtain a first tracking feature map by encoding the plurality of image foreground key frames through a first convolutional neural network using a three-dimensional convolution kernel; An area of ​​interest extraction module, used for respectively passing each of the multiple image foreground key frames through an object detection network to obtain multiple face areas of interest; A second convolutional encoding module, used for obtaining a second tracking feature map by encoding the plurality of face regions of interest through a second convolutional neural network using a three-dimensional convolution kernel; A feature distribution correction module, used for correcting the first tracking feature map and the second tracking feature map to obtain a corrected first tracking feature map and a corrected second tracking feature map respectively; A feature fusion module, used for fusing the corrected first tracking feature map and the corrected second tracking feature map to obtain a classification feature map; as well as A state detection result generation module, used for passing the classification feature graph through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the object to be monitored has abnormal behavior; The image key frame extraction module is used to extract the multiple image key frames from the monitoring video at a predetermined sampling frequency; The image key frame extraction module is used to extract the multiple image key frames from the monitoring video using a difference frame method; The first convolutional encoding module is used to use each layer of the first convolutional neural network to perform three-dimensional convolution processing, pooling processing and nonlinear activation processing based on the three-dimensional convolution kernel on the input data in the forward transfer of the layer, so as to output the first tracking feature map from the last layer of the first convolutional neural network, wherein the input of the first layer of the first convolutional neural network is the multiple image foreground key frames; The target detection network is a target detection network based on no anchor window, and the target detection network based on no anchor window includes YOLOv1, FCOS, CenterNet, CornerNet, ExtremeNet, and RepPoints; The feature distribution correction module comprises: a first weight calculation unit, configured to calculate a frugal decomposition encouragement factor of the first tracking feature map as a weighting coefficient as a first weight, wherein the frugal decomposition encouragement factor of the first tracking feature map as a weighting coefficient is related to a weighted sum of a natural exponential function raised to a power of a negative value of an eigenvalue at each position in each feature matrix of the first tracking feature map with a feature matrix as a weighting unit; a second weight calculation unit, configured to calculate a frugal decomposition encouragement factor of the second tracking feature map as a weighting coefficient as a second weight, wherein the frugal decomposition encouragement factor of the second tracking feature map as a weighting coefficient is related to a weighted sum of a natural exponential function raised to a power of a negative value of an eigenvalue at each position in each feature matrix of the first tracking feature map with a feature matrix as a weighting unit; A first correction unit, configured to weight the feature values ​​of each position in the first tracking feature map with the first weight to obtain the corrected first tracking feature map; and The second correction unit is used to weight the feature value of each position in the second tracking feature map with the second weight to obtain the corrected second tracking feature map.

2. The office personnel status detection system based on image recognition according to claim 1 is characterized in that: The first weight calculation unit is further used to calculate the frugal decomposition encouragement factor of the first tracking feature map as a weighting coefficient as a first weight using the following formula, wherein the formula is: Where w1 is the first weight, is the eigenvalue of the first tracking feature map, τ1 is a hyperparameter, ReLU represents the ReLU activation function, and ||·||2 represents the second norm of the vector.

3. The office personnel status detection system based on image recognition according to claim 2 is characterized in that: The second weight calculation unit is further used to calculate the frugal decomposition encouragement factor of the second tracking feature map as a weighting coefficient as a second weight using the following formula, wherein the formula is: Where w2 is the second weight, is the eigenvalue of the second tracking feature map, τ2 is a hyperparameter, ReLU represents the ReLU activation function, and ||·||2 represents the second norm of the vector.

4. The office personnel status detection system based on image recognition according to claim 3 is characterized in that: The state detection result generating module is further used for: The classification feature map is processed by the classifier using the following formula to generate a classification result, wherein the formula is: softmax{(W n , B n ):...:(W1,B1)|Project(F)}, where Project(F) represents projecting the classification feature map into a vector, W1 to W n is the weight matrix of each fully connected layer, B1 to B n Represents the bias matrix of each fully connected layer.

5. A detection method using an office personnel status detection system based on image recognition according to any one of claims 1 to 4, characterized in that: include: Obtaining surveillance videos of the monitored object collected by cameras deployed in the office; Extracting a plurality of image key frames from the surveillance video; Extracting a foreground portion of each image key frame in the plurality of image key frames to obtain a plurality of image foreground key frames; The plurality of image foreground key frames are passed through a first convolutional neural network using a three-dimensional convolution kernel to obtain a first tracking feature map; Passing each of the plurality of image foreground key frames through an object detection network to obtain a plurality of face regions of interest; The plurality of face regions of interest are passed through a second convolutional neural network using a three-dimensional convolution kernel to obtain a second tracking feature map; Correcting the first tracking characteristic map and the second tracking characteristic map respectively to obtain a corrected first tracking characteristic map and a corrected second tracking characteristic map; fusing the corrected first tracking feature map and the corrected second tracking feature map to obtain a classification feature map; as well as The classification feature graph is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the object to be monitored has abnormal behavior.

Images