A spatiotemporal joint analysis method of human behavior based on cellular automata

Abstract

The present invention discloses a human behavior spatiotemporal joint analysis method based on the cellular automaton idea. First, the obtained three-dimensional human skeleton posture key points are discretized based on the cellular automaton idea; secondly, the energy value of the three-dimensional human skeleton posture key points and the position change of the human skeleton key points are combined to calculate the energy smoothing value of the three-dimensional human skeleton posture key points; finally, a human spatiotemporal joint analysis deep network is constructed, and the three-dimensional coordinate information extracted frame by frame from the human fall video and the calculated smoothing value of the energy change after the key change of the human body are input into the constructed human behavior spatiotemporal joint analysis network for training to perform human posture detection. The present invention simplifies the behavior analysis technology in the field of machine vision from complex environmental human image recognition processing to the coordinates of the three-dimensional human skeleton key points and the energy change of the human skeleton key points, reducing the computer computing power required for behavior analysis, and has a wide range of application scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of posture detection in the field of machine vision, and specifically relates to a method for spatiotemporal joint analysis of human behavior based on the idea of ​​cellular automata. Background Art

[0002] The research content of human posture recognition is the method of restoring human posture composed of fixed joint points based on the observation of images. Human posture detection requires the positioning of key nodes of the human body in a complex and changing environment. The focus of human recognition includes human body detection and key point positioning. Deep learning and human posture estimation technology are developing rapidly today, and the research on human behavior analysis has also received widespread attention. Relevant researchers have proposed a large number of human posture detection methods for this purpose.

[0003] When conducting target-specific behavior analysis today, most researchers directly process the data of the two-dimensional human skeleton key points obtained by the human posture estimation method. However, the detection of the two-dimensional human skeleton model will have errors in many cases, and the extracted key points will be blocked due to a specific action in the three-dimensional space where the target is located. This will have a considerable impact on the behavior analysis using the detected results, causing errors in the judgment made based on the continuity of the image. Although there are also three-dimensional skeleton and skeleton key point extraction technologies nowadays, compared with the two-dimensional skeleton and skeleton key point extraction technologies, the environmental information in the three-dimensional space increases geometrically compared to the environmental information in the two-dimensional space, and the current more accurate methods for estimating three-dimensional human posture have higher requirements for equipment, such as motion capture methods. This leads to the fact that although the accuracy of three-dimensional human posture estimation will be significantly improved compared to the two-dimensional accuracy, the disadvantages include high research costs, large computing power requirements, and poor timeliness of detection in complex environments. Feature extraction occupies an important position in the human behavior recognition and monitoring system, and its effectiveness has a significant impact on the results of subsequent behavior recognition. Summary of the invention

[0004] In view of the above problems, the present invention provides a method for spatiotemporal joint analysis of human behavior based on the idea of ​​cellular automata. The method first obtains a three-dimensional human skeleton key point model and discretizes it after establishing a coordinate system; secondly, the energy of the human skeleton key points and the energy changes of the human skeleton key points caused by the position changes of the skeleton key points when the human body moves are defined; finally, the four obtained data sequences are processed by the sliding window method, and the constructed spatiotemporal joint analysis network of human behavior is trained, and the trained network can be used for human behavior analysis.

[0005] The present invention provides a method for spatiotemporal joint analysis of human behavior based on the cellular automaton concept, comprising the following steps:

[0006] (1) Using the optical gait capture technology of an infrared camera to obtain a three-dimensional skeleton posture key point model of the human body, and establishing a three-dimensional coordinate system in the projection space of the infrared camera where the three-dimensional skeleton posture key point model of the human body is located, and discretizing the skeleton posture key points in the three-dimensional skeleton posture key point model of the human body according to the idea of ​​cellular automation;

[0007] (2) defining the energy value function of the human skeleton posture key points in the human body three-dimensional skeleton posture key point model and the smoothing value function of the energy change value function of the human skeleton posture key points when the human body moves;

[0008] (3) Constructing a spatiotemporal joint analysis network for human behavior;

[0009] (4) Use the trained human behavior spatiotemporal joint analysis network to perform real-time analysis of human behavior.

[0010] Furthermore, in the step (1), the matplotlib framework in Python is used to establish a three-dimensional coordinate system for the human skeleton posture key points in the projection space of the infrared camera where the human body three-dimensional skeleton posture key point model is located, and the coordinates of all the human skeleton posture key points in the three-dimensional coordinate system are obtained.

[0011] Furthermore, the energy value function of the key points of the human skeleton posture in step (2) is:

[0012] e i (t) = W i ×Z i (t)

[0013] In the formula, e i (t) is the energy value of the i-th human skeleton posture key point in the t-th frame, and the z coordinate value at time t, W i is the energy weight of the key point of the i-th human skeleton posture, Z i (t) is the z coordinate value of the i-th human skeleton posture key point in the t-th frame;

[0014] The smoothing value function of the energy change value of the key points of the human skeleton posture when the human body moves is:

[0015]

[0016] Where ΔE i (t, t+1) is the smoothed value of the energy change of the i-th human skeleton posture key point from the t-th to the t+1-th frame, Δe i (t, t+1) is the energy change value of the key point of the i-th human skeleton posture from the t-th to the t+1-th frame, Δe i (t,t+1)=e i (t+1)-ei (t), e i (t+1) is the energy value of the i-th human skeleton posture key point in the t+1-th frame.

[0017] Furthermore, the human behavior spatiotemporal joint analysis network in step (3) includes four LSTM neural network layers, four Flatten network layers, a feature fusion layer and a fully connected layer; the input of each LSTM neural network layer is respectively an x-coordinate value sequence, a y-coordinate value sequence, a z-coordinate value sequence, and a smoothed value sequence of energy change values ​​of the key points of the human skeleton posture; the output of each LSTM neural network layer is respectively converted into one-dimensional data through a Flatten network layer; the four groups of one-dimensional data are fused into a group of four-dimensional data through the feature fusion layer; the group of four-dimensional data passes through the fully connected layer and outputs the classification result.

[0018] Furthermore, when the human behavior spatiotemporal joint analysis network is trained in the step (4), for the monitored human fall video, the x-coordinate value sequence, y-coordinate value sequence, z-coordinate value sequence of the human skeletal posture key points and the smoothed value sequence of the energy change values ​​of the human skeletal posture key points in adjacent frames are obtained frame by frame, and then the sliding window method is used to obtain training data from the four sequences.

[0019] further, Wherein, n is the number of human skeleton posture key points in the human 3D skeleton posture key point model.

[0020] The present invention starts from the two aspects of the temporal change and spatial change of human posture points, and proposes a spatiotemporal joint analysis method of human behavior based on the idea of ​​cellular automata. On the one hand, the three-dimensional skeleton model of the human body is discretized, and a coordinate system is established to extract the three-dimensional coordinates of the key points of the skeleton; on the other hand, the energy concept of each key point of the human body is defined, and the change of the total energy of the human body brought by the key points when the human body moves is defined. The video is analyzed frame by frame, and the three-dimensional coordinate data is extracted. According to the data, the energy of the adjacent frame change of the human body and the smoothed value of the energy are obtained, and these four data are used as four data sequences. After processing them with the sliding window method, the constructed spatiotemporal joint analysis network of human behavior is trained, and the trained network can perform human behavior analysis. The present invention simplifies the behavior analysis technology in the field of machine vision from the complex environment human image recognition processing to the coordinates of the three-dimensional skeleton key points of the human body and the simple situation of the energy change of the key points of the human skeleton, reducing the computer computing power required for behavior analysis, and has a wide range of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The three-dimensional skeleton of the human body when standing;

[0022] Figure 2It is the three-dimensional skeleton of the human body when falling;

[0023] Figure 3 It is a schematic diagram of the deep network model structure;

[0024] Figure 4 It is a schematic diagram of the sliding window method;

[0025] Figure 5 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the invention is described in detail below with reference to the accompanying drawings:

[0027] In one embodiment, Figure 5 As shown, a method for spatiotemporal joint analysis of human behavior based on the idea of ​​cellular automata is provided, comprising the following steps:

[0028] (1) Using the optical gait capture technology of an infrared camera to obtain a three-dimensional skeleton posture key point model of the human body, and establishing a three-dimensional coordinate system in the projection space of the infrared camera where the three-dimensional skeleton posture key point model of the human body is located, and discretizing the skeleton posture key points in the three-dimensional skeleton posture key point model of the human body according to the idea of ​​cellular automation;

[0029] (2) defining the energy value function of the human skeleton posture key points in the human body three-dimensional skeleton posture key point model and the smoothing value function of the energy change value function of the human skeleton posture key points when the human body moves;

[0030] (3) Construct a spatiotemporal joint analysis network for human behavior;

[0031] (4) Use the trained human behavior spatiotemporal joint analysis network to perform real-time analysis of human behavior.

[0032] In one embodiment, optical gait capture technology using an infrared camera is used to obtain Figure 1 The three-dimensional skeleton of the human body when standing as shown Figure 2 The three-dimensional skeleton of a human body falling as shown in the figure shows that, in specific behaviors such as falling, it can be obviously concluded that, according to the change of coordinate axes, the key points of the bones in the head part change the most, followed by the hands and shoulders, less so in the hips, and almost no change in the feet.

[0033] In one embodiment, the matplotlib framework in Python is used to establish a three-dimensional coordinate system for the human skeletal posture key points of the human three-dimensional skeleton in the above two situations in the projection space of the infrared camera where the human three-dimensional skeletal posture key point model is located in three-dimensional space, and the coordinates of 33 skeletal key points in the coordinate system are obtained.

[0034] In one embodiment, the energy value function defining the key points of the human skeleton posture is:

[0035] e i (t) = W i ×Z i (t)

[0036] Among them, e i (t) is the energy value of the i-th human skeleton posture key point in the t-th frame, and the z coordinate value at time t, Z i (t) is the z coordinate value of the i-th human skeleton posture key point in the t-th frame; W i is the energy weight of the i-th human skeleton posture key point, indicating the importance of each human skeleton posture key point in different scenes, and the specific value satisfies:

[0037]

[0038] Wherein, n is the number of human skeleton posture key points in the human 3D skeleton posture key point model.

[0039] In one embodiment, the smoothing value function defining the energy change value of the human skeleton posture key points during human body movement is:

[0040]

[0041] Among them, ΔE i (t, t+1) is the smoothed value of the energy change of the i-th human skeleton posture key point from the t-th to the t+1-th frame, Δe i (t, t+1) is the energy change value of the key point of the i-th human skeleton posture from the tth to t+1th frame, Δe i (t,t+1)=e i (t+1)-e i (t), e i (t+1) is the energy value of the i-th human skeleton posture key point in the t+1-th frame.

[0042] In one embodiment, Figure 3 As shown, first, for the monitored human fall video, the x-coordinate value sequence, y-coordinate value sequence, z-coordinate value sequence of the human skeleton posture key points in the human three-dimensional skeleton posture key point model and the smoothed value sequence of the energy change value of the human skeleton posture key points in adjacent frames are obtained frame by frame; secondly, the four different sequences are input into four LSTM neural networks respectively; thirdly, the output of each LSTM neural network is converted into one-dimensional data through a Flatten network to generate four groups of one-dimensional data; thirdly, the four groups of one-dimensional data are feature fused to obtain a group of four-dimensional data, and the group of four-dimensional data is classified after passing through the fully connected layer.

[0043] In one embodiment, Figure 4 As shown, for the x-coordinate value sequence, y-coordinate value sequence, z-coordinate value sequence of the key points of the human skeleton posture and the smoothed value sequence of the energy change values ​​of the key points of the human skeleton posture in adjacent frames, the sliding window method is used to slide the window to the right one by one according to the frame sequence, and the data in the window is used as training data to train the human behavior spatiotemporal joint analysis network.

[0044] The human behavior spatiotemporal joint analysis method based on the cellular automaton idea provided by the present invention first discretizes the key points of the obtained three-dimensional human skeleton posture based on the cellular automaton idea; secondly, the energy value of the key points of the human skeleton posture in the proposed three-dimensional human skeleton model and the position change of the key points of the human skeleton are combined to calculate the smooth value of the energy change after the position change of the key points of the human skeleton; finally, a human spatiotemporal joint analysis deep network is constructed, and the three-dimensional coordinate information extracted frame by frame from the human fall video and the calculated smooth value of the energy change after the key changes of the human skeleton are input into the constructed human behavior spatiotemporal joint analysis network for training to perform human posture detection. The present invention simplifies the behavior analysis technology in the field of machine vision from the complex environment human image recognition processing to the coordinates of the three-dimensional human skeleton key points and the energy change of the key points of the human skeleton, reducing the computer computing power required for behavior analysis, and has a wide range of application scenarios.

[0045] It should be pointed out that the description of the above embodiments is only used to help understand the method of the present application and its core idea. For ordinary technicians in this technical field, several improvements and modifications can be made to the present application without departing from the principles of the present application. These improvements and modifications are also within the scope of protection of the claims of the present application.

Claims

1. A spatiotemporal joint analysis method of human behavior based on the idea of ​​cellular automata, characterized by: The following steps are involved: (1) Using the optical gait capture technology of an infrared camera to obtain a three-dimensional skeleton posture key point model of the human body, and establishing a three-dimensional coordinate system in the projection space of the infrared camera where the three-dimensional skeleton posture key point model of the human body is located, and discretizing the skeleton posture key points in the three-dimensional skeleton posture key point model of the human body according to the idea of ​​cellular automation; (2) defining the energy value function of the human skeleton posture key points in the human three-dimensional skeleton posture key point model and the smoothing value function of the energy change value function of the human skeleton posture key points when the human body moves; (3) Construct a spatiotemporal joint analysis network for human behavior; (4) Analyze human behavior in real time using the trained human behavior spatiotemporal joint analysis network; The energy value function of the key points of the human skeleton posture in step (2) is: e i (t)=W i ×Z i (t) In the formula, e i (t) is the energy value of the i-th human skeleton posture key point in the t-th frame, and the z coordinate value at time t, W i is the energy weight of the key point of the i-th human skeleton posture, Z i (t) is the z coordinate value of the i-th human skeleton posture key point in the t-th frame; The smoothing value function of the energy change value of the key points of the human skeleton posture when the human body moves is: In the formula, ΔE i (t, t+1) is the smoothed value of the energy change of the i-th human skeleton posture key point from the t-th to the t+1-th frame, Δe i (t, t+1) is the energy change value of the key point of the i-th human skeleton posture from the t-th to the t+1-th frame, Δe i (t,t+1)=e i (t+1)-e i (t), e i (t+1) is the energy value of the i-th human skeleton posture key point in the t+1-th frame.

2. The method for spatiotemporal joint analysis of human behavior based on cellular automata according to claim 1 is characterized in that: In the step (1), the matplotlib framework in Python is used to establish a three-dimensional coordinate system for the human skeleton posture key points in the projection space of the infrared camera where the human skeleton posture key point model is located, and the coordinates of all the human skeleton posture key points in the three-dimensional coordinate system are obtained.

3. The method for spatiotemporal joint analysis of human behavior based on cellular automata according to claim 1 is characterized in that: The human behavior spatiotemporal joint analysis network in step (3) includes four LSTM neural network layers, four Flatten network layers, a feature fusion layer and a fully connected layer; the input of each LSTM neural network layer is respectively an x-coordinate value sequence, a y-coordinate value sequence, a z-coordinate value sequence, and a smoothed value sequence of energy change values ​​of the key points of the human skeleton posture; the output of each LSTM neural network layer is respectively converted into one-dimensional data through a Flatten network layer; the four groups of one-dimensional data are fused into a group of four-dimensional data through the feature fusion layer; the group of four-dimensional data is output as a classification result through the fully connected layer.

4. The method for spatiotemporal joint analysis of human behavior based on cellular automata according to claim 1 is characterized in that: When training the human behavior spatiotemporal joint analysis network in the step (4), for the monitored human fall video, the x-coordinate value sequence, y-coordinate value sequence, z-coordinate value sequence of the human skeletal posture key points and the smoothed value sequence of the energy change values ​​of the human skeletal posture key points in adjacent frames are obtained frame by frame, and then the sliding window method is used to obtain training data from the four sequences.

5. The method for spatiotemporal joint analysis of human behavior based on cellular automata according to claim 4 is characterized in that: Wherein, n is the number of human skeleton posture key points in the human 3D skeleton posture key point model.

Images