Human body posture estimation method based on dynamic graph convolutional network

Abstract

A human body posture estimation method based on a dynamic graph convolutional network belongs to the field of human body posture estimation, and comprises the following steps: step 1, simplifying a human body structure in a video, and constructing a human body skeleton key point structure chart; 2, defining a graph structure to represent a spatial relationship between human joints; thirdly, a graph structure is defined to represent the motion relevance between the joints of the human body; 4, constructing a dynamic graph convolutional network by using dynamic space-time graph convolution; 5, 3D human body posture behavior data enhancement is carried out; and 6, constructing human body behavior data in the virtual scene by using the 3D human body posture data. According to the method, the modeling capability of the complex relation between the joints is improved, the robustness of the model is improved, and complex behaviors are recognized more accurately.

Description

Technical Field

[0001] The present invention belongs to the field of human body posture estimation, and in particular relates to a human body posture estimation method based on a dynamic graph convolutional network. Background Art

[0002] Human behavior recognition is a challenging task in computer vision. According to the different input data processing methods, human behavior recognition can be divided into methods based on RGB video and 3D skeleton data. Based on RGB video, human-related motion information is usually obtained by capturing information such as time, space and optical flow in the video to achieve human behavior recognition. However, these RGB video-based recognition methods have some limitations, especially in indoor monitoring environments. Since the camera is usually installed at an angle above the entire space, it is very difficult to obtain human behavior information using the RGB image of the original video. In addition, in this environment, the problem of time and space information loss may also occur, further increasing the difficulty of human behavior recognition.

[0003] In contrast, the human skeleton sequence information obtained from the video is more robust and can eliminate the interference of background information. There is a strong correlation between the various joints of the human skeleton model, which can provide rich body structure information, and the timing information can be obtained through the skeleton data of continuous frames. Skeleton data has a lower dimension, which reduces the complexity of the input data to improve the efficiency of recognition. The method based on skeleton data not only improves the accuracy of recognition, but also can better cope with the challenges in various complex environments, especially in special scenes such as elevator cars. Human behavior recognition through skeleton models can not only overcome the limitations of traditional RGB video methods, but also provide more reliable and accurate recognition results, thus playing an important role in computer vision applications.

[0004] Thanks to the development of various neural networks, neural networks can be used to efficiently process low-dimensional skeleton data. Related research is as follows: Ail et al. used 3D posture estimation to process video data and constructed a dual 3D posture baseline deep neural network for the recognition of passenger fall behavior in elevators; Lan et al. first extracted the characteristics of elevator passenger behavior through the spatial domain and the temporal domain respectively, and then constructed a two-stream convolutional network to identify abnormal behavior of elevator passengers. However, these supervised video detection methods perform well in laboratory environments, but the generalization ability of the trained models is weak and it is difficult to use them for human behavior videos from different perspectives. This is mainly because the training of these models requires a large amount of training data, and each training data set is balanced.

[0005] Therefore, it is very important to select a reasonable method to enhance the data of abnormal human behavior. Existing research methods usually use image synthesis and direct posture generation to enhance posture. Secondly, in some specific scenarios, research can obtain corresponding data sets by organizing experimenters to perform specific actions. The data sets generated by these methods can be used in models of human behavior recognition to improve the performance of the model.

[0006] Although the above methods can generate human behavior data for human behavior recognition, their research has made certain contributions in the field of human behavior data enhancement. However, the current challenges in the research of human behavior data enhancement include: (1) The behavior data generated by the model training method is often unreasonable, or too simple for model training, and cannot effectively improve the accuracy of the model. (2) Some dangerous scenarios or abnormal behaviors that are not conducive to personal safety are difficult to obtain new data sets by organizing personnel experiments, especially in elevator cars. Once abnormal behaviors that endanger the operation of the elevator are performed, it may cause equipment damage and casualties. (3) The generated human behavior data is not diverse enough and cannot effectively improve the generalization ability of the model. Summary of the invention

[0007] In order to overcome the shortcomings of the prior art, the present invention provides a human posture estimation method based on a dynamic graph convolutional network (Dynamic Graph Convolutional Network-based Behavior data Augmentation, DGCN-BA), constructs a dynamic graph convolutional network to capture the spatiotemporal characteristics of human joints and perform 3D human posture estimation, and then the posture enhancer performs two types of enhancement operations to adjust the bone angle and length respectively. In addition, a posture discriminator is introduced to judge the rationality of the enhanced posture sequence to ensure that it can generate diverse and reasonable 3D human posture data. Finally, the modeling software is used to construct the behavior of virtual scenes and virtual characters, and the enhanced data of human behavior is generated for human behavior recognition.

[0008] The technical solution adopted by the present invention to solve its technical problem is: A human body posture estimation method based on a dynamic graph convolutional network, the method comprising the following steps: The first step is to simplify the human body structure in the video and construct a structural diagram of the key points of the human skeleton; Step 2: Define a graph structure to represent the spatial relationship between human joints; Step 3: Define a graph structure to represent the motion correlation between the joints of the human body; Step 4: Use dynamic spatiotemporal graph convolution to build a dynamic graph convolution network. The process is as follows: Step (4.1) uses a 2D pose estimation method to obtain a 2D human pose sequence from the original video image sequence as the input of the dynamic graph convolutional network; Step (4.2) constructs a dynamic graph convolution module to learn the positional relationship of joints in each frame of human posture data, and further learns the temporal posture relationship in different coordinate systems; The fifth step is to enhance the 3D human posture behavior data; The sixth step is to use 3D human posture data to construct human behavior data in the virtual scene.

[0009] Further, the process of the first step is as follows: Step (1.1) Select representative key points of human skeleton according to the movement characteristics of human body in steps (1.1.1-1.1.2); Step (1.1.1) For the limbs, select the wrist, elbow, shoulder, ankle, knee, and hip joints as key points according to the characteristics of human movement; Step (1.1.2) For the torso part, select the nose, neck, chest and pelvis as key points according to the characteristics of the human body structure; Step (1.2) connects the key points of the human skeleton with straight lines according to the human body structure to form a complete key point structure diagram of the human skeleton.

[0010] Going one step further, the process of the second step is as follows: Step (2.1) Define the spatial graph , ; Each node in the spatial graph Represents the positions of all human joints in a human posture, and the edges in the spatial graph Indicates the connection between two joints; Step (2.2) constructs the corresponding human joint feature matrix for each human posture , ; in, k Indicates the number of the image frame. Each human joint feature matrix includes all the joints in a person's posture, and each joint has a unique number. i , used to distinguish different joints, and each joint uses a feature vector Indicates that this feature vector contains the position information of the joint. n Represents the total number of joints in a human pose; Step (2.3) obtains the spatial relationship between each joint in the human body posture, and calculates the Euclidean distance between each joint and other joints in turn; Step (2.4) marks the closest joints and builds a local relationship network between joints. ; in, KNN refer to K The nearest neighbor algorithm, according to step (2.3), obtains the Euclidean distance between joints and finds the i of N the closest joint; in, N Indicates the number of other joints associated with the target joint; Step (2.5) constructs a dynamic affinity matrix for each joint , used to indicate the strength of the relationship between this joint and other joints, ; Among them, when the joint j belong When i and joints j There is relevance; in, j Represents the joint with the target in a human pose i Other joint numbers that are not the same; Step (2.6) Design weight mechanism To measure each joint j With target joint i The importance of ; in, Represents a nonlinear mapping; Step (2.6) multiplies the dynamic affinity matrix by the weight mechanism element by element to obtain the weighted affinity matrix , ; in, Represents the element-by-element multiplication symbol; Step (2.7) performs steps (2.7.1-2.7.3) to construct dynamic spatial graph convolution , .

[0011] Preferably, the process of step (2.7) is: Step (2.7.1) converts the weighted affinity matrix and the joint feature matrix Perform weighted calculations, combining the structural relationships and features between joints; Step (2.7.2) builds a trainable parameter matrix , linearly combine the matrices obtained in step (2.7.1) to enhance the information transfer between different joint points; Step (2.7.3) passes the result obtained in step (2.7.2) into the ReLu activation function , the result after linear combination is introduced into nonlinear characteristics.

[0012] Furthermore, the process of the third step is as follows: Step (3.1) constructs a temporal graph to describe the spatial relationship between human joints in each frame. ; in, Contains the human joint feature matrix of three adjacent frames in a human pose ( , , ), It includes the human joint connection between two adjacent frames, which involves two affinity matrices, namely, k and k + 1 The forward affinity matrix between the frame k and k- 1, the backward affinity matrix between; Step (3.2) in frame k- 1 and k+ 1. Use K The nearest neighbor method calculates the feature changes of each joint in adjacent frames; Step (3.3) calculates the Euclidean distance between these feature changes, marking and frame k Middle target joint i Joints with similar movements , , ; ; Step (3.4) constructs the forward affinity matrix that changes over time and the backward affinity matrix , ; ; Step (3.5) constructs a weight mechanism for adjacent frames , Indicates the importance of each joint. ; ; in, and They are all nonlinear mappings; Step (3.6) concatenates the joint feature vectors of two adjacent frames as the input of the weight mechanism, and then uses the fully connected layer for learning; Step (3.7) will , and , After element-by-element multiplication, we get the weighted forward affinity matrix and the backward affinity matrix , ; ; Step (3.8) performs steps (3.8.1-3.8.3) to construct dynamic time graph convolution , .

[0013] Preferably, the process of step (3.8) is: Step (3.8.1) weights the forward weighted affinity matrix and the joint feature matrix of the next frame, and the backward weighted affinity matrix and the joint feature matrix of the previous frame respectively; Step (3.8.2) constructs three trainable parameter matrices , and , linearly transform the two matrices obtained in step (3.8.1) and the joint feature matrix of the current frame respectively; Step (3.8.3) adds the results calculated in step (3.8.2) and then passes them into the ReLu activation function , introducing nonlinear characteristics.

[0014] The process of step (4.1) is: Step (4.1.1) sets the number of frames of the original video data to 60 frames; Step (4.1.2) uses the CPN detection step (4.1.1) to detect the video processed and obtain the coordinates of the skeleton key points of the human body in the video; Step (4.1.3) connects the coordinate points identified in each frame of the image in step (4.1.2) according to the human skeleton key point structure diagram in step (1.2) to obtain 2D human posture sequence data.

[0015] The process of step (4.2) is: Step (4.2.1) maps the 2D human posture obtained in step (4.1.3) into a 128-dimensional posture feature; Step (4.2.2) processes the 128-dimensional posture features obtained in step (2.7) through dynamic spatial graph convolution to capture the positional features of human joints; Step (4.2.3) then obtains the motion characteristics of human joints through the dynamic time graph convolution obtained in step (3.8); Step (4.2.4) adds a NonLocal block after each of the two dynamic graph convolutions to improve the ability to capture global information; In step (4.2.5), the dynamic graph convolution module is stacked four more times to enhance the feature extraction capability of the network and finally obtain the estimated 3D human posture.

[0016] The process of step five is as follows: Step (5.1) uses the posture features extracted by the dynamic graph convolutional network to regress the parameters of the two posture enhancements , They are used to change the joint angles and bone lengths of the human body to obtain a new posture; in, is a dimension of 3 ( n -1) Residual matrix used to change joint angles, is a dimension of 1 ( n -1) Ratio vector used to change the length of bones; Step (5.2) performs steps (5.2.1-5.2.2) to enhance the 3D human posture data. The process is: Step (5.2.1) decomposes the joint vector into the joint direction matrix and the joint length matrix ; in, It only contains the position and direction information of each joint. It only contains the length information of each joint; Step (5.2.2) uses , Transform the joint angles and joint lengths of the original human body posture to generate new 3D human body posture data. ; ; in, is the transformed joint orientation matrix, is the transformed bone length matrix; Step (5.3) constructs a posture discriminator for judging the rationality of posture data. The process is: Step (5.4) sets reasonable bone angle and length parameters to judge the input data of the posture discrimination module; Among them, the basis for setting the bone angle and length parameters is provided by the biodynamic structure; Among them, the biodynamic structure provides the physical and mechanical characteristics of the human body in motion, and based on this information, the limit angles of each joint of the human body and the relative size of a bone in the entire human posture are obtained; Step (5.5) feeds back the output result of the posture discrimination module to the posture enhancement module to guide the posture enhancement module to generate more reasonable posture data.

[0017] Preferably, the process of step (5.3) is: Step (5.3.1) transforms the 3D human pose obtained in step (4.2.5) by layered transformation n Joints converted to n - 1 column bone matrix ; Among them, hierarchical transformation is to perform a series of geometric or vector operations according to the hierarchical relationship of the human bone structure to convert the joint points into bone representations; in, Each column vector of represents each bone, which contains the length and direction information of the bone; Step (5.3.2) Construct bone angle matrix , ; in, The diagonal elements of can be expressed as the length features of each bone, and the non-diagonal elements are expressed as the angle features between different bones. T Represents the transpose operation of a matrix; Step (5.3.3) converts the human skeleton feature matrix Orthogonalize to get ; Step (5.3.4) constructs the bone angle change matrix for adjacent frames respectively , , used to calculate the angle change of bones in different frames, ; ; Among them, the diagonal elements of the bone angle change matrix and Indicates the angle change of the same joint in adjacent frames; Step (5.3.5) Construct the input of the posture discrimination module , .

[0018] The process of step six is ​​as follows: Step (6.1) using modeling software to build virtual simulation scenes and virtual characters; Step (6.2) creates a bone structure for the virtual character; Among them, the skeleton structure consists of multiple skeleton nodes, each skeleton node represents a joint, and the skeletons are connected to each other through a hierarchical structure; Among them, the bone structure of the virtual character is consistent with the hierarchy and relationship of the joints in the 3D human posture data; Step (6.3) binds the geometric model of the virtual character to the bone structure obtained in step (6.2); Step (6.4) applies the 3D human pose data sequence generated in step 5 to the virtual character; Step (6.5) sets up a virtual camera, records the actions of the virtual character, and obtains human behavior data.

[0019] The beneficial effects of the present invention are mainly manifested in: (1) The dynamic graph convolutional network adaptively adjusts the graph structure according to the input data, so as to better capture the dynamic changes of the human body's joint relationships; this adaptive feature enables the model to automatically optimize the graph structure when facing different types of behaviors or posture changes, thereby improving the modeling ability of complex relationships between joints; (2) The enhanced data retains the spatial and temporal structural characteristics of the original behavior, and at the same time expands the possible behavior patterns through changes in different angles or behavior speeds, which helps the model better cope with the changing behavior scenarios in reality during training and improves the robustness of the model; (3) In applications that require high-precision behavior analysis, such as multi-person interaction scenarios and sports competitions, the enhancement method can generate rich data samples to help the model more accurately identify complex behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a diagram of the key points of the human skeleton, where 0 represents the nose, 1 represents the neck, 2 represents the right shoulder, 3 represents the right elbow, 4 represents the right wrist, 5 represents the left shoulder, 6 represents the left elbow, 7 represents the left wrist, 8 represents the chest, 9 represents the pelvis, 10 represents the right hip joint, 11 represents the right knee, 12 represents the right ankle, 13 represents the left hip joint, 14 represents the left knee, and 15 represents the left ankle; Figure 2 It is a dynamic graph convolutional network structure diagram; Figure 3 It is the 3D human body pose estimation result map; Figure 4 This is the 3D human pose estimation result using DGCN-BA; Figure 5 This is the behavior recognition result diagram. DETAILED DESCRIPTION

[0021] The present invention will be further described below in conjunction with the accompanying drawings.

[0022] Reference Figure 1~Figure 5 , a human posture estimation method based on dynamic graph convolutional network, comprising the following steps: The first step is to simplify the human body structure in the video and construct a key point structure diagram of the human skeleton. The process is as follows: Step (1.1) selects representative key points of human skeleton according to the movement characteristics of human body. The process is as follows: Step (1.1.1) For the limbs, select the wrist, elbow, shoulder, ankle, knee, and hip joints as key points according to the characteristics of human movement; Step (1.1.2) For the torso part, select the nose, neck, chest and pelvis as key points according to the characteristics of the human body structure; Step (1.2) connect the key points of the human skeleton with straight lines according to the human body structure to form a complete key point structure diagram of the human skeleton; The second step is to define a graph structure to represent the spatial relationship between human joints. The process is as follows: Step (2.1) Define the spatial graph ,, ; Each node in the spatial graph Represents the positions of all human joints in a human posture, and the edges in the spatial graph Indicates the connection between two joints; Step (2.2) constructs the corresponding human joint feature matrix for each human posture , ; in, k Indicates the number of the image frame. Each human joint feature matrix includes all the joints in a person's posture, and each joint has a unique number. i , used to distinguish different joints, and each joint uses a feature vector Indicates that this feature vector contains the position information of the joint. n Represents the total number of joints in a human pose; Step (2.3) obtains the spatial relationship between each joint in the human body posture, and calculates the Euclidean distance between each joint and other joints in turn; Step (2.4) marks the closest joints and builds a local relationship network between joints. ; in, KNN refer to K The nearest neighbor algorithm, a classification algorithm, obtains the Euclidean distance between joints according to step (2.3) and finds the joint i of N the closest joint; in, N Indicates the number of other joints associated with the target joint; Step (2.5) constructs a dynamic affinity matrix for each joint , used to indicate the strength of the relationship between this joint and other joints, ; Among them, when the joint j belong When i and joints j There is relevance; in, j Represents the joint with the target in a human pose i Other joint numbers that are not the same; Step (2.6) Design weight mechanism To measure each joint j With target joint i The importance of ; in, Represents a nonlinear mapping, where the human joint feature vectors are connected in series in groups of two as the input of the weight mechanism, and then the fully connected layer is used to learn the weights between these joints. Finally, the nonlinear mapping uses the ReLu activation function for nonlinear combination and outputs the results to determine which joints are more important; Step (2.6) multiplies the dynamic affinity matrix by the weight mechanism element by element to obtain the weighted affinity matrix , ; in, Represents the element-by-element multiplication symbol; Step (2.7) performs steps (2.7.1-2.7.3) to construct dynamic spatial graph convolution , ; Step (2.7.1) converts the weighted affinity matrix and the joint feature matrix Perform weighted calculations, combining the structural relationships and features between joints; Step (2.7.2) builds a trainable parameter matrix , linearly combine the matrices obtained in step (2.7.1) to enhance the information transfer between different joint points; Step (2.7.3) passes the result obtained in step (2.7.2) into the ReLu activation function , the result after linear combination is introduced into nonlinear characteristics; Among them, the introduction of nonlinear characteristics means that the relationship between input and output no longer satisfies the linear equation, and changes in input lead to complex and disproportionate changes in output; The third step is to define a graph structure to represent the motion correlation between the joints of the human body. The process is as follows: Step (3.1) constructs a temporal graph to describe the spatial relationship between human joints in each frame. ; in, Contains the human joint feature matrix of three adjacent frames in a human pose ( , , ), It includes the human joint connection between two adjacent frames, which involves two affinity matrices, namely, k and k + 1 The forward affinity matrix between the frame k and k- 1, the backward affinity matrix between; Step (3.2) in frame k- 1 and k+ 1. Use K The nearest neighbor method calculates the feature changes of each joint in adjacent frames; Step (3.3) calculates the Euclidean distance between these feature changes, marking and frame k Middle target joint i Joints with similar movements , , ; ; Step (3.4) constructs the forward affinity matrix that changes over time and the backward affinity matrix , ; ; Step (3.5) constructs a weight mechanism for adjacent frames , Indicate the importance of each joint:

[0023]

[0024] in, and They are all nonlinear mappings; Step (3.6) concatenates the joint feature vectors of two adjacent frames as the input of the weight mechanism, and then uses the fully connected layer for learning; Step (3.7) will , and , After element-by-element multiplication, we get the weighted forward affinity matrix and the backward affinity matrix , ; ;

[0025] Step (3.8) performs steps (3.8.1-3.8.3) to construct dynamic time graph convolution , ; Step (3.8.1) weights the forward weighted affinity matrix and the joint feature matrix of the next frame, and the backward weighted affinity matrix and the joint feature matrix of the previous frame respectively; Step (3.8.2) constructs three trainable parameter matrices , and , linearly transform the two matrices obtained in step (3.8.1) and the joint feature matrix of the current frame respectively; Step (3.8.3) adds the results calculated in step (3.8.2) and then passes them into the ReLu activation function , introducing nonlinear characteristics; Step 4: Use dynamic spatiotemporal graph convolution to build a dynamic graph convolution network. The process is as follows: Step (4.1) performs steps (4.1.1-4.1.3) to obtain a 2D human pose sequence using a 2D pose estimation method on the original video image sequence as the input of the dynamic graph convolutional network. The process is: Step (4.1.1) sets the number of frames of the original video data to 60 frames; Step (4.1.2) uses Cascaded Pyramid Network (CPN) to detect the video processed in step (4.1.1) and obtain the coordinates of the skeleton key points of the human body in the video; Among them, CPN is a 2D human pose estimation method that is used to identify the human skeleton joints in the video and output the two-dimensional coordinates of each joint point; Step (4.1.3) connects the coordinate points identified in each frame of the image in step (4.1.2) according to the human skeleton key point structure diagram in step (1.2) to obtain 2D human posture sequence data; Step (4.2) performs steps (4.2.1-4.2.5) to construct a dynamic graph convolution module, which is used to learn the position relationship of joints in each frame of human posture data, and further learn the time posture relationship in different coordinate systems. The process is: Step (4.2.1) maps the 2D human posture obtained in step (4.1.3) into a 128-dimensional posture feature; Among them, by mapping 2D human posture to a high-dimensional feature space of 128 dimensions, the model extracts more accurate posture information, thereby improving the ability to distinguish between different postures; Step (4.2.2) processes the 128-dimensional posture features obtained in step (2.7) through dynamic spatial graph convolution to capture the positional features of human joints; Step (4.2.3) then obtains the motion characteristics of human joints through the dynamic time graph convolution obtained in step (3.8); Step (4.2.4) adds a NonLocal block after each of the two dynamic graph convolutions to improve the ability to capture global information; Among them, NonLocal is a functional module that captures the global relationship between different joints through global operations; Step (4.2.5) repeats the dynamic graph convolution module for another 4 times to enhance the feature extraction capability of the network and finally obtain the estimated 3D human posture; The fifth step is to enhance the 3D human posture behavior data. The process is as follows: Step (5.1) uses the posture features extracted by the dynamic graph convolutional network to regress the parameters of the two posture enhancements , They are used to change the joint angles and bone lengths of the human body to obtain a new posture; in, is a dimension of 3 ( n -1) Residual matrix used to change joint angles, is a dimension of 1 ( n -1) Ratio vector used to change the length of bones; Step (5.2) performs steps (5.2.1-5.2.2) to enhance the 3D human posture data. The process is: Step (5.2.1) decomposes the joint vector into the joint direction matrix and the joint length matrix ; in, It only contains the position and direction information of each joint. It only contains the length information of each joint; Step (5.2.2) uses , Transform the joint angles and joint lengths of the original human body posture to generate new 3D human body posture data. ; ; in, is the transformed joint orientation matrix, is the transformed bone length matrix; Step (5.3) performs steps (5.3.1-5.3.5) to construct a posture discriminator for judging the rationality of posture data. The process is: Step (5.3.1) transforms the 3D human pose obtained in step (4.2.5) by layered transformation n Joints converted to n - 1 column bone matrix ; Among them, hierarchical transformation is to perform a series of geometric or vector operations according to the hierarchical relationship of the human bone structure to convert the joint points into bone representations; in, Each column vector of represents each bone, which contains the length and direction information of the bone; Step (5.3.2) Construct bone angle matrix , ;

[0026] in, The diagonal elements of can be expressed as the length features of each bone, and the non-diagonal elements are expressed as the angle features between different bones. T Represents the transpose operation of a matrix; Step (5.3.3) converts the human skeleton feature matrix Orthogonalize to get ; Step (5.3.4) constructs the bone angle change matrix for adjacent frames respectively , , used to calculate the angle change of bones in different frames, ; ;

[0027] Among them, the diagonal elements of the bone angle change matrix and Indicates the angle change of the same joint in adjacent frames; Step (5.3.5) Construct the input of the posture discrimination module , ;

[0028] Step (5.4) sets reasonable bone angle and length parameters to judge the input data of the posture discrimination module; Among them, the basis for setting the bone angle and length parameters is provided by the biodynamic structure; Among them, the biodynamic structure provides the physical and mechanical characteristics of the human body in motion, and based on this information, the limit angles of each joint of the human body and the relative size of a bone in the entire human posture are obtained; Step (5.5) feeds back the output result of the posture discrimination module to the posture enhancement module to guide the posture enhancement module to generate more reasonable posture data; The sixth step is to use 3D human posture data to construct human behavior data in the virtual scene. The process is as follows: Step (6.1) Use Unity3d software to build virtual simulation scenes and virtual characters; Among them, Unity3d is a multi-platform comprehensive development tool that can create interactive content such as 3D video, architectural visualization, real-time 3D animation, etc. Step (6.2) creates a bone structure for the virtual character; Among them, the skeleton structure consists of multiple skeleton nodes, each skeleton node represents a joint, and the skeletons are connected to each other through a hierarchical structure; Among them, the bone structure of the virtual character is consistent with the hierarchy and relationship of the joints in the 3D human posture data; Step (6.3) binds the geometric model of the virtual character to the bone structure obtained in step (6.2); Step (6.4) applies the 3D human pose data sequence generated in step 5 to the virtual character; Step (6.5) sets up a virtual camera, records the actions of the virtual character, and obtains human behavior data.

[0029] Figure 1 It is a diagram of the key points of the human skeleton, which simplifies the human body structure into 16 key points of bones, including nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, chest, pelvis, right hip joint, right knee, right ankle, left hip joint, left knee and left ankle.

[0030] Figure 2 This is a diagram of the structure of a dynamic graph convolution network. The input of the dynamic graph convolution network is a 2D human posture sequence obtained by estimating the 2D posture of a video image sequence. The posture sequence is then input into a dynamic spatiotemporal graph convolution unit for feature extraction. Each unit includes a dynamic spatial graph convolution and a dynamic temporal graph convolution. A NonLocal block is added after both graph convolutions. Finally, the ReLu activation function is used to introduce nonlinearity and another NonLocal block is added at the end of the unit to enhance the overall posture relationship. In addition, the dynamic graph convolution unit is stacked 4 times to enhance the performance of the network.

[0031] The implementation scheme of this embodiment is as follows: Step 1: 10 3D human pose estimation methods and 3 human behavior recognition methods are given as follows: GraFormer: This method combines the advantages of graph convolutional network and transformer model, which can better capture the complex spatial relationship between human skeleton nodes. It is a 3D human posture estimation model.

[0032] PoseFormer: This method captures temporal information and spatial relationships and combines it with Transformer to better handle posture changes in videos and show extremely high results in scenarios with multi-frame input. It is a 3D human posture estimation model.

[0033] HDFormer: This method combines high-resolution feature modeling and a multi-level Transformer structure to achieve high accuracy and high robustness in multi-target and dynamic scenes. It is a 3D human pose estimation model.

[0034] MixSTE: This method focuses on efficiently processing spatial and temporal information. By mixing and encoding spatial and temporal information, it can capture the dynamic changes between different frames and the spatial relationship of human joints. It is a 3D human posture estimation model.

[0035] STCFormer: This method introduces spatial and temporal context features to better capture the dynamic changes of human posture and the details of complex movements. It is a 3D human posture estimation model that can maintain high accuracy and robustness in complex and changing scenes.

[0036] D3DP: This method can accurately estimate the 3D posture of the human body in motion capture and video analysis through efficient spatial and temporal feature modeling. It is particularly suitable for processing multiple targets and scenes with fast posture changes.

[0037] SemGCN: This method builds a semantic graph to enhance the semantic association between joint nodes and combines it with a graph convolutional network to improve the model's estimation accuracy from 2D to 3D posture.

[0038] Baseline: This method is a 3D human pose estimation model based on convolutional neural networks.

[0039] ST-GCN: This method represents human skeleton data as a spatiotemporal graph structure and captures the spatial and temporal relationships of joints through graph convolution for 3D human pose estimation.

[0040] DGCN-BA: A 3D human pose estimation method proposed in this paper. It combines dynamic graph convolutional networks and data enhancement methods to flexibly obtain the spatiotemporal characteristics of human behavior and realize 3D human pose estimation in various complex scenes.

[0041] CNN: This method processes and analyzes frames in the video stream based on a convolutional neural network (CNN). CNN can effectively extract image features for human behavior recognition.

[0042] LSTM-CNN: This method combines convolutional neural networks and long short-term memory networks (LSTM) to learn the spatiotemporal characteristics of human behavior in videos, and finally introduces the attention mechanism to realize human behavior recognition.

[0043] YOLO-CNN: This method combines an object detection model (YOLO) and a temporal convolutional neural network (3D-CNN) to accurately detect and locate human bodies in videos and extract the spatiotemporal features of human bodies, ultimately achieving human behavior recognition.

[0044] Step 2: The experimental data used in the present invention are derived from three large-scale public human behavior video datasets and an elevator passenger behavior video dataset collected by elevator monitoring.

[0045] Human3.6M is a large public dataset for 3D human pose estimation, which collects the actions of 11 actors in 7 different indoor scenes, all of which are recorded by 4 cameras; Among them, 3DHP is a 3D human pose estimation dataset consisting of indoor and outdoor scenes. It consists of 1.3 million frames of images, recording 8 activities of 8 actors from 14 camera angles; Among them, MuPoTS-3D is a 3D dataset for multi-person pose estimation, which collects 8,000 video frames of 20 real scenes, each of which contains the activities of 2-3 people; The elevator passenger behavior video dataset contains four common abnormal behaviors of elevator passengers, including door grabbing, door kicking, falling, and violent incidents. Each behavior contains 200 video clips, each video clip is about 10 seconds, 30 frames per second; Step 3: Define the evaluation indicators. The process is as follows: In order to verify the prediction performance of each model, the present invention uses the following evaluation indicators for experiments: The mean per-joint position error (MPJPE) between the ground truth and the network-estimated 3D poses, and the MPJPE after aligning the predicted joints with the true joints (ProcrustesAligned-MPJPE, PA-MPJPE). Both are in mm.

[0046] Behavior recognition accuracy Accuracy The definition is as follows, ;

[0047] in, TP Indicates the number of samples classified correctly, Total Represents the overall number of samples used for classification.

[0048] Step 4: Analyze and compare the results: Compared with other models, DGCN-BA can flexibly model and aggregate multi-level features to cope with complex human postures, and the model can obtain more accurate joint position estimates.

[0049] like Figure 3 As shown in the figure, in the 3D human pose estimation task of 15 behaviors (guiding, discussing, eating, greeting, calling, taking pictures, posing, shopping, sitting down, sitting, smoking, waiting, walking the dog, walking, and walking together) on the Human3.6M dataset, DGCN-BA achieved a low level of MPJPE for each behavior, and the average MPJPE reached 38.7, while the average MPJPE of the other 7 models (i.e. GraFormer, PoseFormer, HDFormer, MixSTE, STCFormer, D3DP) were 51.8, 44.3, 42.6, 40.9, 40.5, and 40.0, respectively, which are significantly better than other models. This result shows that DGCN-BA can better obtain 3D human poses than other models.

[0050] At the same time, this example selects three 3D pose estimation models and performs 3D pose estimation in two different situations: using DGCN-BA and not using DGCN-BA.

[0051] like Figure 4 As shown in the figure, compared on the 3DHP and MuPoTS-3D datasets, the average MPJPE and PA-MPJPE of the three models (i.e., SemGCN, Baseline, and ST-GCN) are 110.3, 74.2, 160.3, and 112.0, and 102.5, 72.9, 129.6, and 78.7, respectively. After using the 3D posture data generated by DGCN-BA for assistance, the average MPJPE and PA-MPJPE of the three models are reduced to 99.7, 66.5, 148.7, and 102.6, 91.8, 67.4, 113.5, and 69.3, and 90.9, 63.1, 118.9, and 76.7, respectively. This result shows that DGCN-BA can generate reasonable and diverse 3D human posture data and can effectively improve the performance of various models.

[0052] Finally, the DGCN-BA was used to enhance the elevator passenger behavior video dataset. Specifically, Unity3d software was used to simulate and model the elevator scene and passengers in the video, and the behavior of the virtual character was constructed in the simulation scene. 200 video clips were recorded for each type of abnormal behavior data of elevator passengers. Each video clip was about 10 seconds, and 30 frames were set per second. That is, the enhanced dataset doubled the abnormal behavior dataset, and at the same time, in order to ensure the balance of positive and negative samples, the normal behavior data was also doubled in the enhanced data.

[0053] like Figure 5 As shown in the figure, by comparing the original elevator passenger behavior video dataset and the enhanced elevator passenger behavior video dataset, the average accuracy of the three models (i.e., CNN, LSTM-CNN, and YOLO-CNN) in four common elevator passenger behaviors (falling, door grabbing, door kicking, and violent incidents) are 92.7%, 94.4%, and 96.7%, respectively. After using the enhanced dataset for learning and training, the average recognition accuracy of the three models is increased to 94.2%, 96.2%, and 97.7%, respectively. The results show that the behavior data generated by DGCN-BA can be used for training of human behavior recognition and can effectively improve the performance of the recognition model.

[0054] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A human posture estimation method based on dynamic graph convolutional network, characterized in that: The method comprises the following steps: The first step is to simplify the human body structure in the video and construct a structural diagram of the key points of the human skeleton; Step 2: Define a graph structure to represent the spatial relationship between human joints; Step 3: Define a graph structure to represent the motion correlation between the joints of the human body; Step 4: Use dynamic spatiotemporal graph convolution to build a dynamic graph convolution network. The process is as follows: Step (4.1) uses a 2D pose estimation method to obtain a 2D human pose sequence from the original video image sequence as the input of the dynamic graph convolutional network; Step (4.2) constructs a dynamic graph convolution module to learn the positional relationship of joints in each frame of human posture data, and further learns the temporal posture relationship in different coordinate systems; The fifth step is to enhance the 3D human posture behavior data; The sixth step is to use 3D human posture data to construct human behavior data in the virtual scene.

2. The method for estimating human posture based on a dynamic graph convolutional network as claimed in claim 1, characterized in that: The process of the first step is as follows: Step (1.1) Select representative key points of human skeleton according to the movement characteristics of human body in steps (1.1.1-1.1.2); Step (1.1.1) For the limbs, select the wrist, elbow, shoulder, ankle, knee, and hip joints as key points according to the characteristics of human movement; Step (1.1.2) For the torso part, select the nose, neck, chest and pelvis as key points according to the characteristics of the human body structure; Step (1.2) connects the key points of the human skeleton with straight lines according to the human body structure to form a complete key point structure diagram of the human skeleton.

3. The human body posture estimation method based on dynamic graph convolutional network according to claim 1 or 2, characterized in that: The process of the second step is as follows: Step (2.1) Define the spatial graph , ； Each node in the spatial graph Represents the positions of all human joints in a human posture, and the edges in the spatial graph Indicates the connection between two joints; Step (2.2) constructs the corresponding human joint feature matrix for each human posture , ； in, k Indicates the number of the image frame. Each human joint feature matrix includes all the joints in a person's posture, and each joint has a unique number. i , used to distinguish different joints, and each joint uses a feature vector Indicates that this feature vector contains the position information of the joint. n Represents the total number of joints in a human pose; Step (2.3) obtains the spatial relationship between each joint in the human body posture, and calculates the Euclidean distance between each joint and other joints in turn; Step (2.4) marks the closest joints and builds a local relationship network between joints. ； in, KNN refer to K The nearest neighbor algorithm, according to step (2.3), obtains the Euclidean distance between joints and finds the i of N the closest joint; in, N Indicates the number of other joints associated with the target joint; Step (2.5) constructs a dynamic affinity matrix for each joint , used to indicate the strength of the relationship between this joint and other joints, ； Among them, when the joint j belong When i and joints j There is relevance; in, j Represents the joints with the target in a human pose i Other joint numbers that are not the same; Step (2.6) Design weight mechanism To measure each joint j With target joint i The importance of ； in, Represents a nonlinear mapping; Step (2.6) multiplies the dynamic affinity matrix by the weight mechanism element by element to obtain the weighted affinity matrix , ； in, Represents the element-by-element multiplication symbol; Step (2.7) Construct dynamic spatial graph convolution , 。 4. The method for estimating human posture based on a dynamic graph convolutional network as claimed in claim 3, characterized in that: The process of step (2.7) is: Step (2.7.1) converts the weighted affinity matrix and the joint feature matrix Perform weighted calculations, combining the structural relationships and features between joints; Step (2.7.2) builds a trainable parameter matrix , linearly combine the matrices obtained in step (2.7.1) to enhance the information transfer between different joint points; Step (2.7.3) passes the result obtained in step (2.7.2) into the ReLu activation function , the result after linear combination is introduced into nonlinear characteristics.

5. The method for estimating human posture based on a dynamic graph convolutional network as claimed in claim 1 or 2, characterized in that: The process of the third step is as follows: Step (3.1) constructs a temporal graph to describe the spatial relationship between human joints in each frame. ； in, Contains the human joint feature matrix of three adjacent frames in a human pose ( , , ), It includes the human joint connection between two adjacent frames, which involves two affinity matrices, namely, k and k+ 1 The forward affinity matrix between the frame k and k- 1, the backward affinity matrix between; Step (3.2) in frame k- 1 and k+ 1. Use K The nearest neighbor method calculates the feature changes of each joint in adjacent frames; Step (3.3) calculates the Euclidean distance between these feature changes, marking and frame k Middle target joint i Joints with similar movements , , ； ； Step (3.4) constructs the forward affinity matrix that changes over time and the backward affinity matrix , ； ； Step (3.5) constructs a weight mechanism for adjacent frames , Indicates the importance of each joint. ； ； in, and They are all nonlinear mappings; Step (3.6) concatenates the joint feature vectors of two adjacent frames as the input of the weight mechanism, and then uses the fully connected layer for learning; Step (3.7) will , and , After element-by-element multiplication, we get the weighted forward affinity matrix and the backward affinity matrix , ； ； Step (3.8) performs steps (3.8.1-3.8.3) to construct dynamic time graph convolution , 。 6. The method for estimating human posture based on a dynamic graph convolutional network as claimed in claim 5, characterized in that: The process of step (3.8) is: Step (3.8.1) weights the forward weighted affinity matrix and the joint feature matrix of the next frame, and the backward weighted affinity matrix and the joint feature matrix of the previous frame respectively; Step (3.8.2) constructs three trainable parameter matrices , and , linearly transform the two matrices obtained in step (3.8.1) and the joint feature matrix of the current frame respectively; Step (3.8.3) adds the results calculated in step (3.8.2) and then passes them into the ReLu activation function , introducing nonlinear characteristics.

7. The method for estimating human posture based on a dynamic graph convolutional network according to claim 1 or 2, characterized in that: The process of step (4.1) is: Step (4.1.1) sets the number of frames of the original video data to 60 frames; Step (4.1.2) uses the CPN detection step ( 4.1.1) The video processed in the above step is used to obtain the coordinates of the key points of the human skeleton in the video; Step (4.1.3) connects the coordinate points identified in each frame of the image in step (4.1.2) according to the human skeleton key point structure diagram in step (1.2) to obtain 2D human posture sequence data; The process of step (4.2) is: Step (4.2.1) maps the 2D human posture obtained in step (4.1.3) into a 128-dimensional posture feature; Step (4.2.2) processes the 128-dimensional posture features obtained in step (2.7) through dynamic spatial graph convolution to capture the positional features of human joints; Step (4.2.3) then obtains the motion characteristics of human joints through the dynamic time graph convolution obtained in step (3.8); Step (4.2.4) adds a NonLocal block after each of the two dynamic graph convolutions to improve the ability to capture global information; In step (4.2.5), the dynamic graph convolution module is stacked four more times to enhance the feature extraction capability of the network and finally obtain the estimated 3D human posture.

8. The method for estimating human posture based on a dynamic graph convolutional network according to claim 1 or 2, characterized in that: The process of step five is as follows: Step (5.1) uses the posture features extracted by the dynamic graph convolutional network to regress the parameters of the two posture enhancements , They are used to change the joint angles and bone lengths of the human body to obtain a new posture; in, is a dimension of 3 ( n -1) Residual matrix used to change joint angles, is a dimension of 1 ( n -1) Ratio vector used to change the length of bones; Step (5.2) performs steps (5.2.1-5.2.2) to enhance the 3D human posture data. The process is: Step (5.2.1) decomposes the joint vector into the joint direction matrix and the joint length matrix ; in, It only contains the position and direction information of each joint. It only contains the length information of each joint; Step (5.2.2) uses , Transform the joint angles and joint lengths of the original human body posture to generate new 3D human body posture data. ； ； in, is the transformed joint orientation matrix, is the transformed bone length matrix; Step (5.3) constructs a posture discriminator for judging the rationality of posture data. The process is: Step (5.4) sets reasonable bone angle and length parameters to judge the input data of the posture discrimination module; Among them, the basis for setting the bone angle and length parameters is provided by the biodynamic structure; Among them, the biodynamic structure provides the physical and mechanical characteristics of the human body in motion, and based on this information, the limit angles of each joint of the human body and the relative size of a bone in the entire human posture are obtained; Step (5.5) feeds back the output result of the posture discrimination module to the posture enhancement module to guide the posture enhancement module to generate more reasonable posture data.

9. The method for estimating human posture based on dynamic graph convolutional network as claimed in claim 8, characterized in that: The process of step (5.3) is: Step (5.3.1) transforms the 3D human pose obtained in step (4.2.5) by layered transformation n Joints converted to n - 1 column bone matrix ; Among them, hierarchical transformation is to perform a series of geometric or vector operations according to the hierarchical relationship of the human bone structure to convert the joint points into bone representations; in, Each column vector of represents each bone, which contains the length and direction information of the bone; Step (5.3.2) Construct bone angle matrix , ； in, The diagonal elements of can be expressed as the length features of each bone, and the non-diagonal elements are expressed as the angle features between different bones. T Represents the transpose operation of a matrix; Step (5.3.3) converts the human skeleton feature matrix Orthogonalize to get ; Step (5.3.4) constructs the bone angle change matrix for adjacent frames respectively , , used to calculate the angle change of bones in different frames, ； ； Among them, the diagonal elements of the bone angle change matrix and Indicates the angle change of the same joint in adjacent frames; Step (5.3.5) constructs the input of the posture discrimination module , 。 10. The method for estimating human posture based on dynamic graph convolutional network according to claim 1 or 2, characterized in that: The process of step six is ​​as follows: Step (6.1) using modeling software to build virtual simulation scenes and virtual characters; Step (6.2) creates a bone structure for the virtual character; Among them, the skeleton structure consists of multiple skeleton nodes, each skeleton node represents a joint, and the skeletons are connected to each other through a hierarchical structure; Among them, the bone structure of the virtual character is consistent with the hierarchy and relationship of the joints in the 3D human posture data; Step (6.3) binds the geometric model of the virtual character to the bone structure obtained in step (6.2); Step (6.4) applies the 3D human pose data sequence generated in step 5 to the virtual character; Step (6.5) sets up a virtual camera, records the actions of the virtual character, and obtains human behavior data.

Images