A method, device, equipment and medium for detecting human behavior type

Abstract

The present application provides a method, device, equipment and medium for detecting the behavior type of a person. The method includes: for each person, performing scene analysis based on a pre-constructed probabilistic knowledge graph and the person's human posture vector to determine the amount of information corresponding to the person's performance behavior within a target time window; determining the feature vector of the person within the target time window based on the person's information amount and human posture vector; finally, inputting the feature vector into a pre-trained behavior classification model to determine the behavior type of the person within the target time window. Through the method and device, the behavior classification model is used to detect the feature vector and determine the behavior type of the person, thereby improving the efficiency of personnel behavior detection in the conference room scene and solving the problem of high conference room monitoring costs in the prior art.

Description

Technical Field

[0001] The present application relates to the technical field of personnel behavior detection, and in particular to a method, a detection device, a equipment and a medium for detecting the type of personnel behavior. Background Art

[0002] Meeting rooms are important places for people to carry out their daily work, and people are paying more and more attention to the safety of meeting rooms. In some important meeting occasions, supervisors often need to know the activities of all participants in a timely manner, including whether there are physical conflicts, people falling, whether there are illegal behaviors such as recording and filming, whether the venue is illegally invaded, etc., so as to prompt security personnel to take timely measures.

[0003] Considering that the above anomalies are mainly caused by abnormal behavior of personnel, the detection of abnormal behavior of personnel in conference rooms can reduce the cost and increase the efficiency of conference room monitoring, making conference rooms safer, more efficient and more confidential. However, the existing indoor monitoring system still requires a lot of manual participation to realize the recognition of various behaviors, and it is impossible to take into account the monitoring of multiple conference rooms at the same time due to limited monitoring manpower. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a method, a detection device, an equipment and a medium for detecting the type of personnel behavior, so as to solve the problem of high cost of conference room monitoring in the prior art and improve the efficiency of personnel behavior detection in the conference room scenario.

[0005] In a first aspect, an embodiment of the present application provides a method for detecting a person's behavior type, the detection method comprising:

[0006] Extracting multiple frames of scene monitoring images within a target time window from the monitoring video of the target monitoring scene;

[0007] Extracting a human body posture vector of each person within the target time window from multiple frames of scene monitoring images;

[0008] For each person, scene analysis is performed based on a pre-constructed probabilistic knowledge graph and the person's human posture vector to determine the amount of information corresponding to the person's performance behavior within the target time window; wherein the probabilistic knowledge graph includes a plurality of area nodes divided by the scene monitoring image of the target monitoring scene, a person node under each area node, and a plurality of skeleton nodes under each person node, which are used to describe the prior probability distribution of the spatial change of the person in the target monitoring scene;

[0009] Determine a feature vector of the person within the target time window according to the information amount of the person and the human posture vector;

[0010] The feature vector is input into a pre-trained behavior classification model to determine the behavior type of the person within the target time window.

[0011] Furthermore, extracting the human body posture vector of each person in the target time window from the multi-frame scene monitoring image includes:

[0012] Performing human body monitoring on multiple frames of scene monitoring images, determining each person in each frame of scene monitoring images, and marking the human body tracking frame corresponding to each person in each frame of scene monitoring images;

[0013] Target tracking is performed based on the human tracking frame corresponding to each person in each frame of the scene monitoring image, and the motion trajectory of each person in multiple frames of the scene monitoring image is determined;

[0014] The posture data of the human image corresponding to the human tracking frame in each frame of the scene monitoring image is extracted to obtain the posture features of multiple skeleton points of each person in each frame of the scene monitoring image to form a human posture vector of each person.

[0015] Furthermore, the probabilistic knowledge graph is constructed through the following steps:

[0016] Acquire a historical surveillance video of the target surveillance scene, and extract a sample human body posture vector of each person from multiple frames of sample scene surveillance images of the historical surveillance video;

[0017] Defining the original structure of the probabilistic knowledge graph based on the characteristics of the target monitoring scene;

[0018] By using the original structure of the probabilistic knowledge graph, the sample human posture vector of each person in each frame of the sample scene monitoring image is statistically analyzed to obtain a probability model of each prior probability distribution.

[0019] Furthermore, the probability type of the prior probability distribution includes at least one of the following: a prior probability distribution of human body posture vector values, a prior probability distribution of human body posture vector jumps, and a prior probability distribution of a person appearing in each area in the target monitoring scene.

[0020] Furthermore, the probability model of each prior probability distribution is constructed through the following steps:

[0021] Based on the sample human body posture vector of each person in the multi-frame sample scene monitoring image and the Gaussian mixture probability model, the human body posture vector value is modeled to obtain the prior probability distribution model of the human body posture vector value;

[0022] The sample human body posture vector jump of each person in any two consecutive frames of sample scene monitoring images in multiple frames of sample scene monitoring images is counted to obtain a priori probability distribution model of human body posture vector jump;

[0023] The region to which each person belongs in each frame of the sample scene monitoring image is counted, and a priori probability distribution model of the person appearing in each region of the target monitoring scene is generated.

[0024] Furthermore, the amount of information corresponding to the performance behavior of the person within the target time window is calculated by the following steps:

[0025] Substituting the human body posture vector of the person in the target time window into the prior probability distribution model of the human body posture vector value, to obtain a first probability value of the person in each frame of the scene monitoring image;

[0026] For any scene surveillance image in the multiple frames of scene surveillance images that has a previous adjacent scene surveillance image, a human body posture vector of the person in the scene surveillance image and a human body posture vector in a previous frame of scene surveillance image adjacent to the scene surveillance image are input into the prior probability distribution model of human body posture vector jump to obtain a second probability value of the person in the scene surveillance image;

[0027] Substituting the human body posture vector of the person in the target time window into the prior probability distribution model of the person appearing in each area of ​​the target monitoring scene, to obtain a third probability value of the person in each frame of the scene monitoring image;

[0028] The amount of information corresponding to the performance behavior of the person within the target time window is calculated based on the multiple first probability values, the multiple second probability values, and the multiple third probability values.

[0029] Furthermore, the behavior classification model is trained by the following steps:

[0030] Performing behavior type labeling on normal sample scene surveillance images with normal behavior of personnel and abnormal sample scene surveillance images with abnormal behavior of personnel in the historical surveillance video to obtain training data;

[0031] Inputting the sample human body posture vector of each person in each training data into the probabilistic knowledge graph to obtain the sample information amount corresponding to the training data;

[0032] For each training data, a sample feature vector corresponding to the training data is determined according to the sample information amount corresponding to the training data and the sample human posture vector of each person in the training data, and the sample feature vector is input into the original classification model to determine the behavior type prediction result of each person in the training data;

[0033] The behavior type prediction result of each person in each training data is compared with the behavior type labeling result of each person, and the original classification model is iteratively trained to obtain the behavior classification model.

[0034] In a second aspect, an embodiment of the present application further provides a device for detecting a type of human behavior, the device comprising:

[0035] A monitoring image acquisition module is used to extract multiple frames of scene monitoring images within a target time window from the monitoring video of the target monitoring scene;

[0036] A human posture vector extraction module is used to extract the human posture vector of each person within the target time window from multiple frames of scene monitoring images;

[0037] An information volume calculation module is used to perform scene analysis for each person based on a pre-constructed probabilistic knowledge graph and the person's human posture vector to determine the information volume corresponding to the person's performance behavior within the target time window; wherein the probabilistic knowledge graph includes a plurality of area nodes divided by the scene monitoring image of the target monitoring scene, a person node under each area node, and a plurality of skeleton nodes under each person node, which are used to describe the prior probability distribution of the spatial change of the person in the target monitoring scene;

[0038] A feature vector generation module, used to determine the feature vector of the person within the target time window according to the information amount of the person and the human posture vector;

[0039] The behavior type detection module is used to input the feature vector into a pre-trained behavior classification model to determine the behavior type of the person within the target time window.

[0040] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the method for detecting the type of personnel behavior as described above are performed.

[0041] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for detecting the type of human behavior as described above are executed.

[0042] The embodiments of the present application provide a method, a detection device, a equipment and a medium for detecting the behavior type of a person. First, a multi-frame scene monitoring image within a target time window is extracted from a monitoring video of a target monitoring scene; then, a human posture vector of each person within the target time window is extracted from the multi-frame scene monitoring image; for each person, a scene analysis is performed based on a pre-constructed probabilistic knowledge graph and the human posture vector of the person to determine the amount of information corresponding to the performance behavior of the person within the target time window; a feature vector of the person within the target time window is determined based on the amount of information of the person and the human posture vector; finally, the feature vector is input into a pre-trained behavior classification model to determine the behavior type of the person within the target time window.

[0043] This application determines the amount of information corresponding to the behavior of a person in the target monitoring scene through a probabilistic knowledge graph, and then determines the person's feature vector through the person's body posture vector and the amount of information, and uses a behavior classification model to detect the feature vector to determine the person's behavior type, thereby solving the problem of high conference room monitoring costs in the prior art. In addition, since the pre-constructed probabilistic knowledge graph includes skeleton nodes under each person node, the probabilistic knowledge graph can provide richer personnel features, and the feature vector extracted from the probabilistic knowledge graph is used to enable the subsequent behavior classification model to directly determine the person's behavior type through the features, thereby improving the efficiency of personnel behavior detection in the conference room scene.

[0044] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flowchart of a method for detecting a person's behavior type provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of the structure of a probabilistic knowledge graph provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of the structure of a device for detecting a type of human behavior provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0051] First, the application scenarios to which the present application is applicable are introduced. The present application can be applied to the field of personnel behavior detection technology.

[0052] Meeting rooms are important places for people to carry out their daily work, and people are paying more and more attention to the safety of meeting rooms. In some important meeting occasions, supervisors often need to know the activities of all participants in a timely manner, including whether there are physical conflicts, people falling, whether there are illegal behaviors such as recording and filming, whether the venue is illegally invaded, etc., so as to prompt security personnel to take timely measures.

[0053] Considering that the above anomalies are mainly caused by abnormal behavior of personnel, the detection of abnormal behavior of personnel in conference rooms can reduce the cost and increase the efficiency of conference room monitoring, making conference rooms safer, more efficient and more confidential. However, the existing indoor monitoring system still requires a lot of manual participation to realize the recognition of various behaviors, and it is impossible to take into account the monitoring of multiple conference rooms at the same time due to limited monitoring manpower.

[0054] Based on this, an embodiment of the present application provides a method for detecting personnel behavior types, which improves the efficiency of personnel behavior detection in a conference room scenario and solves the problem of high conference room monitoring costs in the prior art.

[0055] See also Figure 1 , Figure 1 This is a flow chart of a method for detecting a person's behavior type provided in an embodiment of the present application. Figure 1 As shown in, the detection method provided in the embodiment of the present application includes:

[0056] S101. Extracting multiple frames of scene monitoring images within a target time window from a monitoring video of a target monitoring scene.

[0057] Here, the target monitoring scene can be the monitoring scene of a conference room in a company, and the monitoring video is the video captured by the monitoring camera in the conference room. The target time window can be understood as a window that moves forward in the time domain with a specific step size. For example, the length of the target time window is set to n frames. When the length of the target time window is n frames, n frames of scene monitoring images should be extracted from the monitoring video.

[0058] Regarding the above step S101, in a specific implementation, first, a surveillance video captured by a surveillance camera in a target surveillance scene is acquired, and multiple frames of scene surveillance images within a target time window are extracted from the surveillance video.

[0059] S102, extracting a human body posture vector of each person within the target time window from multiple frames of scene monitoring images.

[0060] Here, the human body posture vector refers to the coordinates, rotation angle and other features of the human body skeleton points of each person in the scene monitoring image. Specifically, a 3D human body model containing 24 human body skeleton points is used. Considering that the lower body of the human body will be estimated inaccurately due to occlusion in the scene, only 14 skeleton points of the upper body are taken in this application. Therefore, each person contains 14 3D (rotation angle along the x-axis, rotation angle along the y-axis, rotation angle along the z-axis, in radians) posture data in each frame of the scene monitoring image.

[0061] For the above step S102, in the specific implementation, for each frame of the scene monitoring image, first extract the posture vector of each person in the scene monitoring image, and then flatten the angle of the skeleton point of each person in the continuous n-frame scene monitoring image to obtain the human posture vector of the person in the target time window. Specifically, flatten the angle of the skeleton point of each person in the continuous n-frame scene monitoring image to obtain a vector of n*14*3, denoted as V=[v1,v2,…,v n ], where v i It is a 42*3-dimensional column vector, which is the posture vector of the person in the i-th frame scene monitoring image.

[0062] As an optional embodiment, with respect to the above step S102, extracting the human body posture vector of each person in the target time window from the multi-frame scene monitoring image includes:

[0063] Step 1021 , performing human body monitoring on multiple frames of scene monitoring images, determining each person in each frame of scene monitoring images, and marking a human body tracking frame corresponding to each person in each frame of scene monitoring images.

[0064] For the above step 1021, in the specific implementation, human body monitoring is performed on multiple frames of scene monitoring images. Specifically, multiple frames of scene monitoring images are input into a pre-trained human body detection model to detect each person in each frame of scene monitoring images, and a human body tracking frame corresponding to each person is marked in each frame of scene monitoring images. Here, the human body detection model can be a model such as MTCNN, SSD, YOLO, etc., and this application does not make specific limitations on this.

[0065] Step 1022: Target tracking is performed based on the human body tracking frame corresponding to each person in each frame of the scene monitoring image to determine the motion trajectory of each person in the multiple frames of the scene monitoring image.

[0066] For the above step 1022, during the specific implementation, target tracking is performed according to the human body tracking frame corresponding to each person in each frame of the scene monitoring image. Specifically, each frame of the scene monitoring image is input into a pre-trained human body tracking model to determine the motion trajectory of each person in multiple frames of scene monitoring images. The human body tracking model here can be DeepSort, Kalman filter, particle filter, etc., and this application does not make specific limitations on this. Assuming that there are a total of x people active in the target monitoring scene, then if the tracking is successful, x human target IDs and their motion trajectories in n frames of scene monitoring images will be obtained.

[0067] Step 1023, extracting posture data of the human image corresponding to the human tracking frame in each frame of the scene monitoring image, obtaining posture features of multiple skeleton points of each person in each frame of the scene monitoring image, so as to form a human posture vector of each person.

[0068] For the above step 1023, in the specific implementation, the posture data of the human body image corresponding to the human body tracking frame in each frame of the scene monitoring image is extracted. Specifically, the human body image corresponding to the human body tracking frame is input into a pre-trained human body posture detection model to obtain the posture features of multiple bone points of each person in each frame of the scene monitoring image, and then the posture features of each bone point in each frame of the scene monitoring image are spliced ​​to form a human body posture vector of each person. Here, the human body posture detection model can be OpenPose, AlphaPose, MetaPose, etc., and this application does not make specific limitations on this.

[0069] S103, for each person, scene analysis is performed based on a pre-constructed probabilistic knowledge graph and the person's human posture vector to determine the amount of information corresponding to the person's performance behavior within the target time window.

[0070] It should be noted that the probabilistic knowledge graph is mainly used to describe the probability distribution of entity relationships and human attributes in indoor monitoring scenarios. Its entity definition and relationship structure are consistent with conventional knowledge graphs, and the difference lies mainly in the modeling of the following probability distributions, including: a) the probability of node attribute occurrence; b) the probability of node attribute jump; c) the probability of the occurrence of a relationship between two nodes.

[0071] Specifically, according to the embodiment provided in the present application, the probabilistic knowledge graph includes multiple area nodes divided by the scene monitoring image of the target monitoring scene, a personnel node under each area node, and multiple skeleton nodes under each personnel node, which are used to describe the prior probability distribution of the spatial changes of personnel in the target monitoring scene.

[0072] See also Figure 2 , Figure 2 A schematic diagram of the structure of a probabilistic knowledge graph provided in an embodiment of the present application. Figure 2 As shown, the nodes "Region 1", "Region 2", etc. are all regions divided from the scene monitoring image. For example, Region 1 is the desktop, Region 2 is the upper right side of the desktop, etc., and this application does not make specific limitations on this. If Region 1 is connected to two nodes "Person 1" and "Person 2", it means that visually "Person 1" and "Person 2" are both in Region 1. A person may visually exist in multiple regions at the same time, so a person's nodes will be connected to multiple regions; and a skeleton point generally only appears in one region, so a skeleton node is only connected to one region node.

[0073] For the above step S103, in the specific implementation, for each person, based on the pre-constructed probabilistic knowledge graph and the person's human posture vector, the scene analysis is performed to determine the amount of information corresponding to the performance behavior of the person in the target time window. The specific process of scene analysis is divided into two main parts: spatial feature extraction and temporal feature extraction. Here, the amount of information refers to the amount of information contained in an event, and the probability refers to the possibility of the occurrence of this event. In information theory, the uncertainty of the occurrence of an event is measured by probability, and the amount of information required to eliminate this uncertainty is measured by the amount of information. The higher the probability, the greater the possibility of the event, and the smaller the amount of information contained; the lower the probability, the smaller the possibility of the event, and the greater the amount of information contained. In this way, since there are skeleton nodes of the person in the pre-constructed probabilistic knowledge graph, the probabilistic knowledge graph can provide richer personnel features. After the characteristic vector of the person is determined by the probabilistic knowledge graph, the behavior classification model can directly determine the behavior type of the person through the characteristic vector.

[0074] As an optional embodiment, the probabilistic knowledge graph is constructed through the following steps:

[0075] A: Obtain a historical surveillance video of the target surveillance scene, and extract a sample human body posture vector of each person from multiple frames of sample scene surveillance images of the historical surveillance video.

[0076] For the above step A, in the specific implementation, a historical surveillance video of the target surveillance scene is obtained, and a sample human body posture vector of each person is extracted from multiple frames of sample scene surveillance images of the historical surveillance video. Here, the method of extracting the sample human body posture vector of a person from the sample scene surveillance image is the same as the method of extracting the human body posture vector of a person from the scene surveillance image in the above step S102, and can achieve the same technical effect, which will not be repeated here.

[0077] B: Define the original structure of the probabilistic knowledge graph based on the characteristics of the target monitoring scene.

[0078] For the above step B, in the specific implementation, the original structure of the probabilistic knowledge graph is defined based on the characteristics of the target monitoring scene. Here, according to the embodiment provided by the present application, when the target scene is a conference room scene, in the original structure, the first layer of nodes includes area nodes, the area nodes are connected to personnel nodes, and the personnel nodes are connected to skeleton nodes.

[0079] C: Using the original structure of the probabilistic knowledge graph, statistics are collected on the sample human posture vectors of each person in each frame of the sample scene surveillance image to obtain a probability model for each prior probability distribution.

[0080] Here, according to the embodiment provided by the present application, the probability type of the prior probability distribution includes at least one of the following items: the prior probability distribution of human body posture vector values, the prior probability distribution of human body posture vector jumps, and the prior probability distribution of people appearing in each area in the target monitoring scene.

[0081] Specifically, according to the embodiments provided in this application, the probability model of each prior probability distribution is constructed through the following steps:

[0082] (1) Based on the sample human posture vector of each person in the multi-frame sample scene monitoring image and the Gaussian mixture probability model, the human posture vector value is modeled to obtain the prior probability distribution model of the human posture vector value.

[0083] For the above step (1), in the specific implementation, the sample human body posture vector w of each person in the multi-frame sample scene monitoring image is i Perform value statistics to obtain a prior probability distribution P1(w i ), and use a Gaussian mixture probability model with a parameter of 5 to fit this high-dimensional probability distribution, that is, to obtain the prior probability distribution model of the human body posture vector value.

[0084] (2) The sample human body posture vector jumps of each person in any two consecutive frames of sample scene monitoring images in multiple frames of sample scene monitoring images are statistically analyzed to obtain a priori probability distribution model of human body posture vector jumps.

[0085] For the above step (2), in the specific implementation, the sample human posture vector jump of each person in any two consecutive frames of sample scene monitoring images in the multi-frame sample scene monitoring image is counted, that is, the value in the current frame sample scene monitoring image is w j In the case of a frame of sample scene monitoring image, the value is w i The probability of the change of the attitude vector value is obtained by the prior probability distribution P2(w i |w j ), and through discretization, we obtain an attribute change probability matrix M, that is, we get the prior probability distribution model of human posture vector jump.

[0086] (3) Counting the region to which each person belongs in each frame of the sample scene monitoring image, and generating a priori probability distribution model of the person appearing in each region of the target monitoring scene.

[0087] For the above step (3), in the specific implementation, the region to which each person belongs in each frame of the sample scene monitoring image is counted, which is recorded as P3 (A i ), that is, area A i The prior probability of a person appearing in the target monitoring scene is used to generate a prior probability distribution model of a person appearing in each area of ​​the target monitoring scene.

[0088] Specifically, with respect to the above step S103, the amount of information corresponding to the performance behavior of the person within the target time window is calculated through the following steps, including:

[0089] Step 1031, substitute the human body posture vector of the person in the target time window into the prior probability distribution model of the human body posture vector value to obtain the first probability value of the person in each frame of the scene monitoring image.

[0090] For the above step 1031, in the specific implementation, the human posture vector of the person in the target time window is substituted into the prior probability distribution model of the human posture vector value to obtain the first probability value of the person in each frame of the scene monitoring image. Specifically, the human posture vector V of the person is regarded as a sampling point of the probability distribution P1, and is substituted into the prior probability distribution model of the human posture vector value to obtain the first probability value of the person in each frame of the scene monitoring image, that is, P1(v1), P1(v2), ..., P1(v n)n probability values. Specifically, by estimating the 3D posture of the human body in the scene monitoring image, the positions and directions of 24 skeleton points are obtained, which are represented by 3 Euler angles. These skeleton points can be represented as: i =(θ xi ,θ yi ,θ zi ),i=1,2,…,24, where i represents the index of the skeleton point. For spatial features, the probability knowledge graph is used to calculate the probability value of each skeleton point. The probability of these skeleton points can be expressed as: P(J i ),i=1,2,…,24.

[0091] Step 1032, for any scene surveillance image in the multiple frames of scene surveillance images that has a previous adjacent scene surveillance image, the human body posture vector of the person in the scene surveillance image and the human body posture vector in the previous frame of scene surveillance image adjacent to the scene surveillance image are input into the prior probability distribution model of the human body posture vector jump to obtain a second probability value of the person in the scene surveillance image.

[0092] In the specific implementation of step 1032, for any scene surveillance image in the multiple scene surveillance images that has a previous adjacent scene surveillance image, the human posture vector of the person in the scene surveillance image and the human posture vector in the previous scene surveillance image adjacent to the scene surveillance image are input into the prior probability distribution model of the human posture vector jump to obtain the second probability value of the person in the scene surveillance image. Specifically, the human posture vector v in the i-th scene surveillance image is input into the prior probability distribution model of the human posture vector jump. i and the human body posture vector v in the j-th frame scene monitoring image j As sampling points of probability distribution P2, we can obtain P2(v2|v1), P2(v3|v2), P2(v4|v3), ..., P1(v n |v n-1 Here, specifically, the extraction of temporal features focuses on analyzing the changes of skeleton points over time. For the temporal feature information, the probability knowledge graph is used to calculate the state transition probability of a skeleton point at two consecutive moments. Assume that the state of the skeleton point at time t is The state at time t+1 is The state transition probability can be expressed by the conditional probability density function, that is, In a given The probability distribution under the condition is expressed as

[0093] Step 1033, substitute the human posture vector of the person in the target time window into the prior probability distribution model of the person appearing in each area in the target monitoring scene, and obtain a third probability value of the person in each frame of the scene monitoring image.

[0094] For the above step 1033, in the specific implementation, the human posture vector of the person in the target time window is substituted into the prior probability distribution model of the person appearing in each area of ​​the target monitoring scene, and the area to which each person belongs in each frame of the scene monitoring image is counted, which is recorded as P3(A i ), that is, area A i The prior probability of a person appearing in .

[0095] Step 1034, calculating the amount of information corresponding to the performance behavior of the person within the target time window based on the multiple first probability values, the multiple second probability values, and the multiple third probability values.

[0096] In the specific implementation of step 1034, the amount of information corresponding to the performance of the person in the target time window is calculated according to the multiple first probability values, the multiple second probability values, and the multiple third probability values. Specifically, the amount of information is calculated by the following formula:

[0097]

[0098]

[0099] or;

[0100]

[0101]

[0102] Among them, I1 and I2 represent the amount of information respectively.

[0103] S104, determining a feature vector of the person within the target time window according to the information amount of the person and the human body posture vector.

[0104] For the above step S104, in the specific implementation, the feature vector of the person in the target time window is determined according to the information amount of the person in the target time window and the human posture vector. Here, specifically, the angle feature and information amount of the skeleton point are flattened according to the skeleton point sequence and time sequence, and m skeleton points are selected from them, J = [0, 1, 2, 3, 4, 5, 6, 9, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23], and set is the Euler angle of the i-th bone point in the t-th frame is the information amount of the i-th bone point in the t-th frame, n is the number of frames considered, then the feature vector of the t-th frame is:

[0105]

[0106] S105, inputting the feature vector into a pre-trained behavior classification model to determine the behavior type of the person within the target time window.

[0107] In the above step S105, in the specific implementation, the feature vector of the person in the target time window is input into the pre-trained behavior classification model to determine the behavior type of the person in the target time window. Here, the behavior type includes normal behavior or any one of multiple abnormal behavior types.

[0108] As an optional embodiment, the behavior classification model is trained through the following steps:

[0109] I: labeling behavior types of normal sample scene surveillance images with normal behaviors of people and abnormal sample scene surveillance images with abnormal behaviors of people in the historical surveillance videos to obtain training data.

[0110] II: Input the sample human posture vector of each person in each training data into the probabilistic knowledge graph to obtain the sample information amount corresponding to the training data.

[0111] For the above steps I to II, in the specific implementation, the normal sample scene monitoring images with normal behavior of personnel and the abnormal sample scene monitoring images with abnormal behavior of personnel in the historical monitoring video are annotated with behavior types to obtain training data. Then, the sample human posture vector of each person in each training data is input into the constructed probabilistic knowledge graph to obtain the sample information amount corresponding to the training data. Here, the method of calculating the sample information amount is the same as the method of calculating the information amount in the above steps, and the same technical effect can be achieved, which will not be repeated here.

[0112] III: For each training data, the sample feature vector corresponding to the training data is determined according to the sample information amount corresponding to the training data and the sample human posture vector of each person in the training data, and the sample feature vector is input into the original classification model to determine the behavior type prediction result of each person in the training data.

[0113] For the above step III, first, combine the time and space attributes and information volume of a small number of labeled samples to form a feature vector, and assign corresponding labels to establish a data set. This data set will be used to train the classifier so that it can classify new input samples as abnormal. In the specific implementation, for each training data, the sample feature vector corresponding to the training data is determined based on the sample information volume corresponding to the training data and the sample human posture vector of each person in the training data. Here, the method for obtaining the sample feature vector is the same as the method for obtaining the feature vector in the above steps, and can achieve the same technical effect, which will not be repeated here. The sample feature vector is then input into the original classification model to determine the behavior type prediction result for each person in the training data. Specifically, as a classifier, the nonlinear model random forest is preferred, and a decision tree can also be used.

[0114] IV: Compare the behavior type prediction result of each person in each training data with the behavior type labeling result of each person, and iteratively train the original classification model to obtain the behavior classification model.

[0115] Regarding the above step IV, in the specific implementation, the behavior type prediction result of each person in each training data is compared with the behavior type labeling result of each person, and the original classification model is iteratively trained to obtain a behavior classification model.

[0116] Here, according to the above steps I to IV, the scene parsing features from all normal behaviors and the scene parsing features of different abnormal behavior types are flattened into feature vectors of the same length, and then combined with the labeled data to train the abnormal classifier. Since the parsing features themselves have sufficient discrimination, only some classic classification models (such as random forests, decision trees, etc.) are needed to complete the classification of different abnormal behaviors.

[0117] The method for detecting the behavior type of a person provided in an embodiment of the present application first extracts a multi-frame scene monitoring image within a target time window from a monitoring video of a target monitoring scene; then extracts a human posture vector of each person within the target time window from the multi-frame scene monitoring image; for each person, scene analysis is performed based on a pre-constructed probabilistic knowledge graph and the human posture vector of the person to determine the amount of information corresponding to the performance behavior of the person within the target time window; a feature vector of the person within the target time window is determined based on the amount of information of the person and the human posture vector; finally, the feature vector is input into a pre-trained behavior classification model to determine the behavior type of the person within the target time window.

[0118] This application determines the amount of information corresponding to the behavior of a person in the target monitoring scene through a probabilistic knowledge graph, and then determines the person's feature vector through the person's body posture vector and the amount of information, and uses a behavior classification model to detect the feature vector to determine the person's behavior type, thereby solving the problem of high conference room monitoring costs in the prior art. In addition, since the pre-constructed probabilistic knowledge graph includes skeleton nodes under each person node, the probabilistic knowledge graph can provide richer personnel features, and the feature vector extracted from the probabilistic knowledge graph is used to enable the subsequent behavior classification model to directly determine the person's behavior type through the features, thereby improving the efficiency of personnel behavior detection in the conference room scene.

[0119] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a human behavior type detection device provided in an embodiment of the present application. Figure 3 As shown in , the detection device 300 includes:

[0120] The monitoring image acquisition module 301 is used to extract multiple frames of scene monitoring images within a target time window from the monitoring video of the target monitoring scene;

[0121] A human posture vector extraction module 302 is used to extract the human posture vector of each person within the target time window from multiple frames of scene monitoring images;

[0122] The information volume calculation module 303 is used to perform scene analysis for each person based on a pre-constructed probabilistic knowledge graph and the person's human posture vector to determine the information volume corresponding to the person's performance behavior within the target time window; wherein the probabilistic knowledge graph includes a plurality of area nodes divided by the scene monitoring image of the target monitoring scene, a person node under each area node, and a plurality of skeleton nodes under each person node, which are used to describe the prior probability distribution of the spatial change of the person in the target monitoring scene;

[0123] A feature vector generating module 304 is used to determine a feature vector of the person in the target time window according to the information amount of the person and the human body posture vector;

[0124] The behavior type detection module 305 is used to input the feature vector into a pre-trained behavior classification model to determine the behavior type of the person within the target time window.

[0125] Furthermore, when the human body posture vector extraction module 302 is used to extract the human body posture vector of each person within the target time window from the multi-frame scene monitoring image, the human body posture vector extraction module 302 is also used to:

[0126] Performing human body monitoring on multiple frames of scene monitoring images, determining each person in each frame of scene monitoring images, and marking the human body tracking frame corresponding to each person in each frame of scene monitoring images;

[0127] Target tracking is performed based on the human tracking frame corresponding to each person in each frame of the scene monitoring image, and the motion trajectory of each person in multiple frames of the scene monitoring image is determined;

[0128] The posture data of the human image corresponding to the human tracking frame in each frame of the scene monitoring image is extracted to obtain the posture features of multiple skeleton points of each person in each frame of the scene monitoring image to form a human posture vector of each person.

[0129] Furthermore, the information volume calculation module 303 is also used to construct the probabilistic knowledge graph through the following steps:

[0130] Acquire a historical surveillance video of the target surveillance scene, and extract a sample human body posture vector of each person from multiple frames of sample scene surveillance images of the historical surveillance video;

[0131] Defining the original structure of the probabilistic knowledge graph based on the characteristics of the target monitoring scene;

[0132] By using the original structure of the probabilistic knowledge graph, the sample human posture vector of each person in each frame of the sample scene monitoring image is statistically analyzed to obtain a probability model of each prior probability distribution.

[0133] Furthermore, the probability type of the prior probability distribution includes at least one of the following: a prior probability distribution of human body posture vector values, a prior probability distribution of human body posture vector jumps, and a prior probability distribution of a person appearing in each area in the target monitoring scene.

[0134] Furthermore, the information amount calculation module 303 is also used to construct a probability model of each prior probability distribution through the following steps:

[0135] Based on the sample human body posture vector of each person in the multi-frame sample scene monitoring image and the Gaussian mixture probability model, the human body posture vector value is modeled to obtain the prior probability distribution model of the human body posture vector value;

[0136] The sample human body posture vector jump of each person in any two consecutive frames of sample scene monitoring images in multiple frames of sample scene monitoring images is counted to obtain a priori probability distribution model of human body posture vector jump;

[0137] The region to which each person belongs in each frame of the sample scene monitoring image is counted, and a priori probability distribution model of the person appearing in each region of the target monitoring scene is generated.

[0138] Furthermore, the information volume calculation module 303 is further used to calculate the information volume corresponding to the performance behavior of the person within the target time window through the following steps:

[0139] Substituting the human body posture vector of the person in the target time window into the prior probability distribution model of the human body posture vector value, to obtain a first probability value of the person in each frame of the scene monitoring image;

[0140] For any scene surveillance image in the multiple frames of scene surveillance images that has a previous adjacent scene surveillance image, a human body posture vector of the person in the scene surveillance image and a human body posture vector in a previous frame of scene surveillance image adjacent to the scene surveillance image are input into the prior probability distribution model of human body posture vector jump to obtain a second probability value of the person in the scene surveillance image;

[0141] Substituting the human body posture vector of the person in the target time window into the prior probability distribution model of the person appearing in each area of ​​the target monitoring scene, to obtain a third probability value of the person in each frame of the scene monitoring image;

[0142] The amount of information corresponding to the performance behavior of the person within the target time window is calculated based on the multiple first probability values, the multiple second probability values, and the multiple third probability values.

[0143] Furthermore, the detection device 300 further includes a model training module, and the model training module is used to train the behavior classification model through the following steps:

[0144] Performing behavior type labeling on normal sample scene surveillance images with normal behavior of personnel and abnormal sample scene surveillance images with abnormal behavior of personnel in the historical surveillance video to obtain training data;

[0145] Inputting the sample human body posture vector of each person in each training data into the probabilistic knowledge graph to obtain the sample information amount corresponding to the training data;

[0146] For each training data, a sample feature vector corresponding to the training data is determined according to the sample information amount corresponding to the training data and the sample human posture vector of each person in the training data, and the sample feature vector is input into the original classification model to determine the behavior type prediction result of each person in the training data;

[0147] The behavior type prediction result of each person in each training data is compared with the behavior type labeling result of each person, and the original classification model is iteratively trained to obtain the behavior classification model.

[0148] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in , the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0149] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 The steps of the method for detecting the type of human behavior in the method embodiment shown, and the specific implementation manner can be found in the method embodiment, which will not be repeated here.

[0150] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the method for detecting the type of human behavior in the method embodiment shown, and the specific implementation manner can be found in the method embodiment, which will not be repeated here.

[0151] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0153] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0155] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0156] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for detecting a person's behavior type, characterized in that: The detection method comprises: Extracting multiple frames of scene monitoring images within a target time window from the monitoring video of the target monitoring scene; Extracting a human body posture vector of each person within the target time window from multiple frames of scene monitoring images; For each person, scene analysis is performed based on a pre-constructed probabilistic knowledge graph and the person's human posture vector to determine the amount of information corresponding to the person's performance behavior within the target time window; wherein the probabilistic knowledge graph includes a plurality of area nodes divided by the scene monitoring image of the target monitoring scene, a person node under each area node, and a plurality of skeleton nodes under each person node, which are used to describe the prior probability distribution of the spatial changes of the person in the target monitoring scene; the amount of information is used to describe the uncertainty of the person's performance behavior within the target time window, the higher the probability of the person's performance behavior within the target time window, the smaller the corresponding amount of information, and the lower the probability of the person's performance behavior within the target time window, the larger the corresponding amount of information; Determine a feature vector of the person within the target time window according to the information amount of the person and the human posture vector; The feature vector is input into a pre-trained behavior classification model to determine the behavior type of the person within the target time window.

2. The detection method according to claim 1, characterized in that: The step of extracting a human body posture vector of each person within the target time window from multiple frames of scene monitoring images includes: Performing human body monitoring on multiple frames of scene monitoring images, determining each person in each frame of scene monitoring images, and marking the human body tracking frame corresponding to each person in each frame of scene monitoring images; Target tracking is performed based on the human tracking frame corresponding to each person in each frame of the scene monitoring image, and the motion trajectory of each person in multiple frames of the scene monitoring image is determined; The posture data of the human image corresponding to the human tracking frame in each frame of the scene monitoring image is extracted to obtain the posture features of multiple skeleton points of each person in each frame of the scene monitoring image to form a human posture vector of each person.

3. The detection method according to claim 1, characterized in that: The probabilistic knowledge graph is constructed through the following steps: Acquire a historical surveillance video of the target surveillance scene, and extract a sample human body posture vector of each person from multiple frames of sample scene surveillance images of the historical surveillance video; Defining the original structure of the probabilistic knowledge graph based on the characteristics of the target monitoring scene; By using the original structure of the probabilistic knowledge graph, the sample human posture vector of each person in each frame of the sample scene monitoring image is statistically analyzed to obtain a probability model of each prior probability distribution.

4. The detection method according to claim 3, characterized in that: The probability type of the prior probability distribution includes at least one of the following: the prior probability distribution of human posture vector values, the prior probability distribution of human posture vector jumps, and the prior probability distribution of people appearing in each area in the target monitoring scene.

5. The detection method according to claim 4, characterized in that: The probability model for each prior probability distribution is constructed by the following steps: Based on the sample human body posture vector of each person in the multi-frame sample scene monitoring image and the Gaussian mixture probability model, the human body posture vector value is modeled to obtain the prior probability distribution model of the human body posture vector value; The sample human body posture vector jump of each person in any two consecutive frames of sample scene monitoring images in multiple frames of sample scene monitoring images is counted to obtain a priori probability distribution model of human body posture vector jump; The region to which each person belongs in each frame of the sample scene monitoring image is counted, and a priori probability distribution model of the person appearing in each region of the target monitoring scene is generated.

6. The detection method according to claim 5, characterized in that: The information amount corresponding to the performance behavior of the person within the target time window is calculated by the following steps, including: Substituting the human body posture vector of the person in the target time window into the prior probability distribution model of the human body posture vector value, to obtain a first probability value of the person in each frame of the scene monitoring image; For any scene surveillance image in the multiple frames of scene surveillance images that has a previous adjacent scene surveillance image, a human body posture vector of the person in the scene surveillance image and a human body posture vector in a previous frame of scene surveillance image adjacent to the scene surveillance image are input into the prior probability distribution model of human body posture vector jump to obtain a second probability value of the person in the scene surveillance image; Substituting the human body posture vector of the person in the target time window into the prior probability distribution model of the person appearing in each area of ​​the target monitoring scene, to obtain a third probability value of the person in each frame of the scene monitoring image; The amount of information corresponding to the performance behavior of the person within the target time window is calculated based on the multiple first probability values, the multiple second probability values, and the multiple third probability values.

7. The detection method according to claim 3, characterized in that: The behavior classification model is trained by the following steps: Performing behavior type labeling on normal sample scene surveillance images with normal behavior of personnel and abnormal sample scene surveillance images with abnormal behavior of personnel in the historical surveillance video to obtain training data; Inputting the sample human body posture vector of each person in each training data into the probabilistic knowledge graph to obtain the sample information amount corresponding to the training data; For each training data, a sample feature vector corresponding to the training data is determined according to the sample information amount corresponding to the training data and the sample human posture vector of each person in the training data, and the sample feature vector is input into the original classification model to determine the behavior type prediction result of each person in the training data; The behavior type prediction result of each person in each training data is compared with the behavior type labeling result of each person, and the original classification model is iteratively trained to obtain the behavior classification model.

8. A device for detecting the type of human behavior, characterized in that: The detection device comprises: A monitoring image acquisition module is used to extract multiple frames of scene monitoring images within a target time window from the monitoring video of the target monitoring scene; A human posture vector extraction module is used to extract the human posture vector of each person within the target time window from multiple frames of scene monitoring images; An information volume calculation module is used to perform scene analysis for each person based on a pre-constructed probabilistic knowledge graph and the person's human posture vector, and determine the information volume corresponding to the person's performance behavior within the target time window; wherein the probabilistic knowledge graph includes a plurality of area nodes divided by the scene monitoring image of the target monitoring scene, a person node under each area node, and a plurality of skeleton nodes under each person node, which are used to describe the prior probability distribution of the spatial change of the person in the target monitoring scene; the information volume is used to describe the uncertainty of the person's performance behavior within the target time window, the higher the probability of the person's performance behavior within the target time window, the smaller the corresponding information volume, and the lower the probability of the person's performance behavior within the target time window, the larger the corresponding information volume; A feature vector generation module, used to determine the feature vector of the person within the target time window according to the information amount of the person and the human posture vector; The behavior type detection module is used to input the feature vector into a pre-trained behavior classification model to determine the behavior type of the person within the target time window.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the method for detecting the type of human behavior as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for detecting the type of human behavior as claimed in any one of claims 1 to 7 are executed.

Images