Personnel attitude estimation method and system based on multi-model graph neural network fusion
Through the multi-model graph neural network fusion method, graph autoencoder and space-time graph attention network are used to accurately capture human postures in indoor environments, solving the occlusion and change challenges in indoor posture estimation, achieving high accuracy and stability Pose tracking.
Patent Information
- Application Number
- CN202410983671.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art faces the challenges of complex environmental occlusion, light and shadow interference, time and space changes in indoor personnel posture estimation, and it is difficult to accurately track and adapt to human posture, and there are identification errors.
The multi-model graph neural network fusion method is adopted to extract key points of the human body through graph autoencoder, the space-time graph attention network captures the space-time correlation, and reduces errors through multi-model ensemble learning, and combines sliding window technology and Bayesian fusion method to improve the model Generalization ability.
Recovering key points of the human body under severe occlusion improves the accuracy and stability of posture estimation, can better track and adapt to the time and space changes of the human body, and reduces recognition errors.
Smart Images

Figure CN118942153A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of posture estimation, and in particular to a method and system for estimating a person's posture based on multi-model graph neural network fusion. Background Art
[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.
[0003] Human posture estimation technology has broad application prospects in improving safety, improving efficiency, and optimizing user experience. For example, in interactive interfaces such as virtual reality and augmented reality, more natural and intuitive interaction methods can be achieved by identifying the user's posture, such as gesture control and virtual entertainment. In addition, posture estimation technology can be used to monitor the daily activities of the elderly or patients with specific diseases (such as Parkinson's disease), thereby providing health monitoring and auxiliary treatment. Human posture estimation technology aims to obtain information about human behavior and movements by analyzing and understanding the posture of the human body in images or videos. It usually includes detecting key points of the human body (such as the head, limbs, etc.) and estimating the position and posture of these key points. In human posture estimation, the two common methods are bottom-up and top-down methods. In the bottom-up method, all possible human key points are first detected in the image or video, and then the algorithm is used to identify and combine the complete human posture based on these detected key points. The top-down method is the opposite. It first detects the overall area of the human body, and then performs key point detection and posture estimation in each detected human body area. Estimating the pose of a person indoors is often more challenging. Compared with open outdoor spaces, indoor environments are usually more closed and contain various obstacles, furniture, and decorations, which may block part or all of the human body, making it difficult for the pose estimation algorithm to obtain complete body information. The bottom-up approach has higher computational complexity and is more robust.
[0004] At present, deep learning-based human posture estimation has made considerable progress. However, the problem of indoor human posture estimation still faces some challenges due to various complex scenes and interference from light and shadow: (1) The indoor environment is complex and changeable. There are various obstacles that may block part of the human body. The collected images not only contain the posture information of the person, but also contain more irrelevant environmental and privacy information, which affects the collection of the person's behavior posture. (2) The behavior posture of a person will change with time and position. There is a correlation between the spatial position of the person's joints and the time point of the person's movement in the collected image data. Conventional algorithms are difficult to dynamically track and adapt to such posture changes. (3) The same posture movement may be performed very differently by different people. The same behavior posture may cause recognition errors due to differences between different people. This requires the posture estimation algorithm to have not only strong robustness, but also high accuracy, and be able to accurately capture every subtle movement of the human body. Summary of the invention
[0005] In response to the above-mentioned defects, the present invention proposes a method and system for personnel posture estimation based on the fusion of multi-model graph neural networks, which can restore the positions of key points of the human body in the case of severe occlusion, and can more accurately capture the spatial relationship and dependence between various parts of the human body, so as to better track and adapt to the temporal and spatial changes of the human body. It also integrates the estimation results of multiple sub-models to reduce the possible deviations and errors of a single model, so that the model can show higher stability when processing various complex postures.
[0006] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:
[0007] The first aspect of the present disclosure provides a method for estimating a person's posture based on multi-model graph neural network fusion, comprising the following steps:
[0008] Collect human posture image sequences, and use sliding windows to divide the collected data sets into multiple sub-data sets;
[0009] The image sequence is constructed into a graph structure, the graph structure is converted into a low-dimensional graph representation, the graph representation is reconstructed, and the key point sequence of the human body is extracted based on the reconstructed image;
[0010] A skeleton sequence graph is constructed based on the key points of the human body, and a spatiotemporal graph is constructed on the skeleton sequence. Multi-layer graph convolution operations are performed on the spatiotemporal graph to obtain a higher-level feature graph, and the posture action category is determined based on the higher-level feature graph.
[0011] The pose estimation results on each sub-dataset are fused to obtain the final pose estimation result.
[0012] As a further implementation method, the image sequence is constructed as a graph structure, specifically:
[0013] Each frame image in the image sequence is divided into multiple parts as graph nodes, the edges of the graph are constructed according to the similarity of each node, and an undirected graph is constructed based on the nodes and edges of the graph.
[0014] As a further implementation method, a key point sequence of the human body is extracted based on the reconstructed image, specifically: the reconstructed image is input into a mask prediction network to obtain a mask image sequence, and the positions of the joint points are extracted from the obtained mask image sequence.
[0015] As a further implementation method, a spatiotemporal graph is constructed on the skeleton sequence, specifically:
[0016] According to the connectivity of the human body structure, the joints in one frame are connected, and then each joint is connected to the same joint in the consecutive frames.
[0017] As a further implementation, the edges of the spatiotemporal graph are composed of two subsets, the first subset is the intra-frame edges, connecting different joints in each frame, and the second subset is the inter-frame edges, connecting the same joints in consecutive frames.
[0018] As a further implementation method, the graph convolution operation includes spatial graph convolution and temporal graph convolution. The spatial graph convolution adopts a spatial attention mechanism to adaptively capture the dynamic correlation between nodes in the spatial dimension, and the temporal graph convolution adopts a temporal attention mechanism to capture the dynamic information between different frames in an image sequence.
[0019] As a further implementation method, the pose estimation results on each sub-dataset are fused to obtain the final pose estimation result, specifically:
[0020] The Bayesian fusion method based on the NUTS algorithm is used to construct the posterior probability distribution of each sub-model, determine the weight of each sub-model, and combine the pose estimation results of multiple sub-models to obtain the final pose estimation.
[0021] The second aspect of the present disclosure provides a personnel posture estimation system based on multi-model graph neural network fusion, comprising:
[0022] The data acquisition module is configured to: acquire a human posture image sequence, and divide the acquired data set into a plurality of sub-data sets using a sliding window;
[0023] The human body key point extraction module is configured to: construct the image sequence into a graph structure, convert the graph structure into a low-dimensional graph representation, reconstruct the graph representation, and extract the human body key point sequence based on the reconstructed image;
[0024] The posture recognition module is configured to: construct a skeleton sequence graph according to the key points of the human body, construct a spatiotemporal graph on the skeleton sequence, perform multi-layer graph convolution operations on the spatiotemporal graph to obtain a higher-level feature graph, and determine the posture action category based on the higher-level feature graph;
[0025] The posture estimation fusion module is configured to: fuse the posture estimation results on each sub-dataset to obtain the final posture estimation result.
[0026] The third aspect of the present disclosure provides a medium on which a program is stored. When the program is executed by a processor, the steps in the method for personnel posture estimation based on multi-model graph neural network fusion described in the first aspect of the present disclosure are implemented.
[0027] The fourth aspect of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for personnel posture estimation based on multi-model graph neural network fusion described in the first aspect of the present disclosure are implemented.
[0028] Compared with the prior art, the present invention has the following beneficial effects:
[0029] The present invention proposes an instance segmentation model based on a graph autoencoder architecture. By decoding and reconstructing the original image, the topological structure and spatial relationship of each part of the human body in the image are effectively preserved, and features are extracted at different scales and directions to achieve accurate segmentation of human body information. When processing the relationship between the human body and occluders, the two-layer graph convolutional network structure extracts useful features from the input data to distinguish between the human body and occluders, and correctly classify and locate the target. It can weaken the interference of external environmental factors and reduce the impact of other objects occluding the human body, while accurately capturing the data of key points of the human body in the image.
[0030] The present invention uses a spatiotemporal graph attention network to capture the spatiotemporal correlation of a person's posture in an image sequence. According to the spatiotemporal attention mechanism, different weights are given to each node, and feature representations with spatiotemporal information are extracted from the original data, which improves the model's ability to process time series data and can better track and adapt to the temporal and spatial changes in the data. It can dynamically track and flexibly adapt to various changes in human postures, and better utilize the spatiotemporal information contained in the data.
[0031] The present invention introduces a multi-model ensemble learning method, uses a sliding window technique to divide the original data set into multiple sub-data sets, and trains multiple spatiotemporal graph attention network sub-models on each subset, which can effectively process time series data. The posterior probability model of each model is constructed by a Bayesian fusion method based on the NUTS algorithm, the corresponding posterior weights are obtained, and the output results of multiple models are integrated, which improves the generalization ability of the model and enables it to maintain a high degree of accuracy in various scenarios. It can reduce the error of the model in the recognition process and reduce the risk of misclassification of behavioral postures due to misjudgment.
[0032] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.
[0034] Figure 1 It is an overall framework flow chart of a method for estimating a person's posture based on multi-model graph neural network fusion disclosed in the present invention;
[0035] Figure 2 is a data processing flow chart of reconstructing features using the graph autoencoder disclosed in the present invention;
[0036] Figure 3 is a flow chart of gesture recognition of the present disclosure;
[0037] Figure 4 It is a computational flow chart of personnel posture estimation using multi-model graph neural network fusion disclosed in the present invention. DETAILED DESCRIPTION
[0038] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.
[0039] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.
[0040] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.
[0041] Embodiment 1
[0042] like Figure 1 As shown, the first embodiment of the present disclosure provides a method for estimating a person's posture based on multi-model graph neural network fusion, which mainly includes the following steps:
[0043] S1 collects human posture image sequences and uses a sliding window to divide the collected data set into multiple sub-data sets;
[0044] S2 constructs the image sequence into a graph structure, converts the graph structure into a low-dimensional graph representation, reconstructs the graph representation, and extracts the key point sequence of the human body based on the reconstructed image;
[0045] S3 constructs a skeleton sequence graph based on the key points of the human body, and constructs a spatiotemporal graph on the skeleton sequence. It performs multi-layer graph convolution operations on the spatiotemporal graph to obtain a higher-level feature graph, and determines the posture action category based on the higher-level feature graph;
[0046] S4 fuses the pose estimation results on each sub-dataset to obtain the final pose estimation result.
[0047] The present invention discloses a bottom-up posture estimation method, which combines graph autoencoder, spatiotemporal graph attention network and multi-model ensemble learning to construct a human posture estimation model.
[0048] In order to accurately extract the key point information of the human body in the image, reduce the interference of the environment, and reduce the impact of objects occluding the human body, the present invention proposes an instance segmentation model based on the graph autoencoder architecture. By extracting the local area of the target object, the attention is focused on the target object. The module first constructs a graph by the original pixel-level feature sequence of the input data, and converts the input image into a low-dimensional graph representation through the encoder part. This graph representation can effectively retain the topological structure and spatial relationship of various parts of the human body in the image. Then, the decoder part reconstructs the original graph according to this graph representation, and finally realizes instance segmentation in a double-layer graph convolutional network structure. The graph autoencoder network can capture the complex spatial relationship of various parts of the human body and extract features at different scales and directions, thereby realizing accurate segmentation of human body information. The double-layer graph convolutional network structure composed of a feature extraction network and a target classification network can effectively identify the relationship between occluders and the human body, distinguish between the human body and occluders, and perform correct target classification and positioning.
[0049] In order to better utilize the spatiotemporal information in the data, dynamically track and adapt to various posture changes of the human body, the present invention uses a spatiotemporal graph attention network to accurately capture the spatiotemporal correlation and dependence between various parts of the human body by introducing an attention mechanism. Through the spatial attention mechanism, each node (corresponding to the joint point of the human body) generates a set of weights by calculating the correlation with other nodes. These weights represent the degree of attention of each node to other nodes and are dynamically adjusted during the calculation process. Similarly, through the time attention mechanism, the weights are dynamically adjusted in the time dimension to capture the correlation of the behavior posture of people in different time slices. The spatiotemporal graph attention network extracts feature representations with spatiotemporal information from the original data by learning a series of image sequences. The spatiotemporal attention mechanism improves the model's ability to process time series data, and can better track and adapt to the temporal and spatial changes in the data.
[0050] In order to reduce the error of model recognition and the possibility of misclassification of behavioral postures, the present invention uses a multi-model ensemble learning method to fuse the posture estimation results output by multiple spatiotemporal graph attention network models, thereby improving the generalization and accuracy of the model. The sliding window technology is used to divide the original data set into multiple sub-datasets, and multiple spatiotemporal graph attention network sub-models are trained on each subset to effectively process the time characteristics in the input image sequence data. A prior distribution is defined for each spatiotemporal graph attention network sub-model by a Bayesian fusion method based on the NUTS algorithm, and the posterior distribution of each sub-model is obtained according to the Bayesian theorem. The NUTS algorithm is further applied to optimize the Bayesian parameters to obtain the optimal posterior weight of each sub-model, and finally the estimation results of each model are Bayesian fused according to their weights to obtain the final integrated estimation result.
[0051] 1. Human key point extraction module based on graph autoencoder
[0052] In the process of detecting the behavior of indoor people, the indoor environment is relatively complex, the objects in the room will block the people, and the collected data images will contain redundant information. These problems will not only cause privacy issues, but also affect the extraction of people's posture information. Therefore, a human key point extraction module is used to extract the local area containing the main part of the human body according to the posture key point information to reduce the interference of irrelevant information on the model.
[0053] This module is mainly composed of graph autoencoders, feature extraction and target classification. The input image sequence is first segmented at the pixel level. The segmented area of each frame in the image sequence is regarded as a node, and the spatial neighbors of each frame are found. The edges of the graph structure are constructed through similarity relationships. After the graph structure of the image is constructed by the above method, it is converted into a low-dimensional graph representation in the autoencoder. The decoder reconstructs the original graph based on this graph representation, and finally implements instance segmentation in the double-layer graph convolutional network structure. The process of reconstructing the graph structure by the graph autoencoder is as follows: Figure 2 shown.
[0054] First, the input image is divided into multiple parts as nodes of the graph, and the edges of the graph are constructed according to the similarity of each node, thus constructing an undirected graph G = (V, E, X). N} represents the set of nodes, N is the number of nodes. E represents the set of edges. The graph structure of graph G can be represented by the adjacency matrix In addition, X is the node attribute matrix associated with the graph, Among them, D is the dimension of the feature.
[0055] The constructed graph G is input into the graph autoencoder network, and the encoder converts the node v i ∈V is embedded into the low-dimensional vector of the encoder Z = f(A,X) Z represents the embedding matrix, which preserves the topological structure A and node information X, making the representation of each node similar to that of its neighboring nodes. The encoder is defined by a variational posterior q(Z|G), which is defined as:
[0056]
[0057] Among them, the parameter μ i and σ i Learned from two graph convolutional layers, the average vector μ = GNN μ (X, A), variance logσ = GNN σ (X,A).
[0058] The encoder network is a two-layer encoder with a symmetric adjacency matrix. The mth encoder layer is defined as:
[0059]
[0060] in, Node attribute matrix Correlation Matrix W (m) is the trainable weight matrix of the mth layer.
[0061] Subsequently, the decoder decodes the representation in the latent space into a reconstruction of the original graph data, that is, to reconstruct the graph structure A and node attributes X, with the goal of preserving the structure and feature information of the original graph as much as possible. The decoder network includes two sub-decoders, including an inner product decoder and a graph filter, for reconstructing A and X.
[0062] Use an inner product decoder to reconstruct the graph structure A:
[0063]
[0064] in is the reconstructed graph structure matrix, Represents the estimate of the graph adjacency matrix after processing by the internal decoder.
[0065] A graph filter is used as a decoder to reconstruct the node attribute X. The mth layer of the decoder is defined as:
[0066]
[0067] in, Node attribute matrix Correlation Matrix From this we can obtain the Laplace determinant of the reconstructed symmetric normalized graph:
[0068]
[0069] in, is a symmetric adjacency matrix, Represents the Laplacian matrix of the reconstructed symmetric normalized graph and the reconstructed node attribute matrix for:
[0070]
[0071] The reconstructed graph structure matrix and node attributes Input to the mask prediction network, which consists of a feature extraction network and a target classification network. The feature extraction network and the target classification network form a two-layer graph convolution network structure, and its graph convolution operation is:
[0072]
[0073] is the learnable weight matrix of the output transformation, and the output features It is composed of node features updated by global information propagation in the entire layer, N = H × W is the number of pixel grids in the ROI area, and K is the feature dimension of each node.
[0074] In the two-layer graph convolutional network composed of the feature extraction network and the target classification network, the input and output of mask prediction are further defined, X ROI represents the input ROI feature, W f Represents the weight of the FCN layer. The ROI feature is first input into the first layer of the network to obtain the output feature Z 0 :
[0075]
[0076] In order to connect the feature extraction network and the target classification network, the output feature Z of the feature extraction network is 0 Add directly to X ROI In the above example, we obtain the fused occlusion-aware feature X f :
[0077]
[0078] X f As the input of the target classification network, the output Z 1 For mask prediction:
[0079]
[0080] Afterwards, the positions of the joint points are extracted from the mask image sequence obtained by the network, and the output Z of the network is used to obtain the position of the joint points. 1 , decode the key points, mark the key points, draw the key points on the image, and visualize them on the mask image. Finally, the output labeled image is blurred to effectively extract the focus area and input into the posture recognition module based on the spatiotemporal graph attention network for further analysis.
[0081] 2. Posture Recognition Module Based on Spatiotemporal Graph Attention Network
[0082] The behavior posture of a person is not static, but changes in various ways with time and location. This change not only reflects the activity status of the person, but also provides important clues for the algorithm to understand their behavioral intentions. In order to dynamically track and adapt to these changes, in this module, a spatiotemporal graph attention network model is used, the spatiotemporal attention mechanism is introduced, the graph neural network is extended to the spatiotemporal graph model, and a skeleton sequence representation for posture recognition is designed.
[0083] First, a human image sequence is obtained from a motion capture device. After the human key point extraction module based on the graph autoencoder extracts the human key point sequence, the data input to the module is a series of frames, each frame has a set of joint coordinates, and the order of body joints is given in the form of two-dimensional coordinates. A spatiotemporal graph with joints as graph nodes and human body structure and time as graph edges is constructed. A multi-layer graph attention mechanism is performed on the input data, and a higher-level feature map is generated on the graph. Then, it is classified into the corresponding action category by a standard Softmax classifier. The entire model is trained in a back-propagation manner. The specific implementation steps of the model are as follows: Figure 3 shown.
[0084] After the human body key points are extracted from the collected human posture images, the focus area is effectively extracted by blurring the background, so that the model can focus more on important features during training and reasoning. Then, a skeleton sequence graph is constructed based on the human body key points. A skeleton sequence is usually represented by the two-dimensional coordinates of each human body joint in each frame. An undirected spatiotemporal graph G = (V, E) is constructed on a skeleton sequence with N joint points and T time slices.
[0085] In this graph, the node set V = {v ti |t=1,K,T,i=1,K,N} includes all joints in a skeleton sequence. As the input of the model, the feature vector F(v ti ) consists of the coordinate vector of the i-th joint on frame t and the estimated confidence. The spatiotemporal graph on the skeleton sequence is constructed in two steps. First, the joints in a frame are connected with edges according to the connectivity of the human body structure. Then, each joint will be connected to the same joint in consecutive frames.
[0086] The edge set E consists of two subsets. The first subset describes the intra-skeleton connections of each frame, denoted by E S = {v ti v tj |(i,j)∈H}, where H is the set of naturally connected human joints. The second subset contains the edges between frames, which are related to E F = {v ti v (t+1)i}Connect the same joints in consecutive frames.
[0087] use represents the value of the nth feature of node i at time t, Represents the value of all features of node i at time t. represents the value of all features of all nodes at time t. τ )∈R N×D×τ represents the values of all features of all nodes over τ time frames.
[0088] In the spatial dimension, a spatial attention mechanism is used to adaptively capture the dynamic correlation between nodes in the spatial dimension:
[0089]
[0090] in, is the input of the rth spatiotemporal block. r-1 is the number of channels of the input data of the rth layer.
[0091] is a learnable parameter, using the RELU function as the activation function. The attention matrix S is dynamically calculated based on the current input of the layer. Then a Softmax function is used to normalize the correlation matrix to obtain the spatial attention weight A s :
[0092]
[0093] S i,j The value of semantically represents the correlation strength between node i and node j. When performing graph convolution, the adjacency matrix A is combined with the spatial attention matrix S∈R N×N Combined to dynamically adjust the influence weights between nodes.
[0094] In the time dimension, there is a correlation between the behavior of people at different moments, and the correlation varies in different situations. Similarly, a temporal attention mechanism is used to adaptively give different weights to the data:
[0095]
[0096] in, is a learnable parameter. The temporal correlation matrix E is determined by the changing input. Normalized with the Softmax function, we get the temporal attention weight A e :
[0097]
[0098] E in E i,j The value semantics of represents the dependency strength between time i and j. The normalized temporal attention weights are directly applied to the input, dynamically adjusting the input by incorporating relevant information:
[0099]
[0100] The temporal attention mechanism is usually used to capture the dynamic information between different frames in an image sequence, while the spatial attention mechanism is used to capture the spatial structure within each frame. Combining them allows the model to better understand the content in a continuous image sequence. By multiplying the attention weights of the temporal and spatial dimensions, the final weighted feature representation is obtained:
[0101] A se =A s eA e (16)
[0102] Where e is the Hadamard product. The final weighted feature representation can be calculated by multiplying the input feature with the fused attention weight.
[0103] The spatiotemporal attention module allows the network to pay more attention to valuable information. The input image is adjusted by the attention mechanism and input into the spatiotemporal convolution module. After feature extraction and attention mechanism adjustment, the feature vector output by the network is passed to the Softmax layer to convert the original output of the network into a probability distribution, thereby deriving the confidence that the input image belongs to each category.
[0104] In general, this module estimates the pose of the collected person image sequence, constructs a spatiotemporal graph on the skeleton sequence, and applies multi-layer graph convolution to gradually generate higher-level feature graphs on the graph. Then, the standard Softmax classifier classifies them into corresponding pose action categories.
[0105] 3. Posture estimation fusion module based on multi-model ensemble learning
[0106] In order to improve the robustness and accuracy of the model, reduce the error of model recognition, and reduce the possibility of misclassification of behavioral postures, an ensemble learning method is used to fuse the estimation results of multiple sub-models, take advantage of the advantages of each sub-model, and reduce the impact of the shortcomings of a single model on the overall performance. Using the Bayesian fusion method based on the NUTS algorithm, by constructing the posterior probability distribution of each sub-model, the weight of each model is determined, and finally the posture estimation results of multiple models are combined to obtain a more accurate and robust final posture estimation.
[0107] This patent adopts a multi-model ensemble learning method, which divides the data set through a sliding window method and constructs multiple spatiotemporal graph attention network sub-models, which can effectively process time series data or tasks that need to consider time dependencies. The sliding window gradually generates training sets and test sets by moving a fixed-size window in the data set, and then trains multiple spatiotemporal graph attention network sub-models within these windows, and finally performs posture estimation through an ensemble method.
[0108] The Bayesian fusion method based on the NUTS algorithm is used to adjust the hyperparameters of the spatiotemporal graph attention network sub-model to find the optimal parameter combination. In the Bayesian fusion method based on the NUTS algorithm, based on the Bayesian reasoning framework, a priori weight is generated for each model to reflect the initial assumptions about the model. As the observed data increases, the NUTS algorithm is further applied to optimize the Bayesian parameters to obtain the optimal posterior weights for each sub-model to reflect the impact of the data on the model. Finally, the final integrated estimation result is obtained by weighted averaging the estimation results of each model according to the posterior weights. The method framework for using this module for fusion estimation of multiple models is as follows: Figure 4 As shown:
[0109] First, sort the data by time field, and use a sliding window to move a fixed-size window in the dataset, each time moving a fixed step length, to generate multiple data subsets containing different time periods. Set the window size to the image sequence data within one day, and the sliding step length to one hour. Starting from the starting position of the dataset, select the data of the window size as the first subset to train and test the model. Then move the window by the sliding step length to generate the next subset, and repeat the training and testing of the model until the entire dataset is covered. Multiple spatiotemporal graph attention network sub-models can be obtained by sliding the data in the window. Finally, the posture estimation results of each trained sub-model are fused through an integration method.
[0110] The steps of combining the posture estimation results of multiple models by the Bayesian fusion method based on the NUTS algorithm are as follows. First, assume that D is the image data of the person's posture behavior collected by the camera, L is the person's posture behavior to be identified, and use multiple spatiotemporal graph attention network models M = [M1, M2, ..., M N ] to train and obtain the corresponding classification results. The Bayesian-based set probability estimation can be expressed as:
[0111]
[0112] Among them, w k is the kth spatiotemporal graph attention network model M k The posterior probability under the given data set D, w k It is also the weight coefficient of the kth model in the Bayesian fusion method. k (L|M k ,D) is the given measured data and the spatiotemporal graph attention network model M k Estimation of human behavior posture results under certain conditions.
[0113] Assume P k (L|M k ,D) is a normal distribution, then:
[0114]
[0115] Among them, L k Based on the measured data and the time graph attention network model M k Estimated personnel behavior posture results. ε M is the graph convolutional network model k The standard deviation of k The calculation error of .
[0116] From equations (17) and (18), we can see that the L-certainty estimate of the Bayesian fusion estimate is the weighted average of all single models, which can be expressed as:
[0117]
[0118] The average Bayesian estimate is a deterministic estimate whose classification accuracy can be compared with that of a single model. The variance of the Bayesian fusion estimate in formula (18) can be calculated using the following formula:
[0119]
[0120] In formula (20), the Bayesian estimation variance is composed of the ensemble estimation and the error of a single model. Using the Bayesian fusion method to estimate the probability of a person’s posture ensemble requires determining the weight coefficients w of different deep learning models. k and the model error σ ε .
[0121] With θ=[w1,w2,K,w N ,σ ε ] represents the unknown parameters to be solved by the Bayesian fusion method, and the maximum likelihood estimation method is used. The log-likelihood function of θ can be expressed as:
[0122]
[0123] Formula (21) involves high-dimensional solution. It is difficult to directly solve θ and it is difficult to directly obtain an analytical solution. This patent uses the NUTS (No-U-Turn Sampler) algorithm to optimize the unknown parameters in the Bayesian fusion method. The NUTS algorithm avoids the need to manually adjust parameters in traditional methods by adaptively adjusting the step size and path length, thereby improving sampling efficiency and adaptability.
[0124] The following are the general steps for Bayesian parameter estimation using NUTS:
[0125] (1) Constructing the objective function: For maximum likelihood estimation, the prior distribution is not considered, so the objective function is the likelihood function.
[0126] (2) Initialize and build initial momentum: randomly select an initial state (parameter value) As a starting point, a momentum variable r is introduced for each parameter with the same dimension as the parameter.
[0127] (3) Calculate the total energy: Combine the parameters and momentum to calculate the total energy of the system:
[0128]
[0129] Where θ is the parameter vector, x is the observed data, p(θ|x) is the posterior probability (or likelihood function), r is the momentum vector, and M is the covariance matrix of the momentum.
[0130] (4) Momentum update: In the NUTS algorithm, the momentum vector is updated according to the rules of Hamiltonian dynamics:
[0131]
[0132] Among them, ε is the step size parameter, is the logarithmic posterior gradient.
[0133] (5) Accept or reject the new state: If the total energy of the new state is lower than the current state, the new state is always accepted. Otherwise, the state with higher total energy is accepted with a certain probability to ensure that the detailed balance condition is met.
[0134] The above steps are executed repeatedly until the termination condition is met, such as reaching the specified number of sampling times or converging to a stable distribution.
[0135] The random samples of unknown parameters obtained by the NUTS algorithm can be considered as samples of the target distribution, thereby avoiding the direct solution of equation (21) and improving the computational efficiency. According to the random samples of the parameters, their statistical characteristics can be estimated respectively, such as the posterior mean, posterior standard deviation or posterior probability density function.
[0136] After calculating the unknown parameters to be found, the posterior probability density function of each model calculated by Bayes' theorem is:
[0137]
[0138] Among them, P(M k ) is the prior probability of the model, P(D|M k ) is the data in model M k The likelihood under , is obtained by the NUTS algorithm.
[0139] After Bayesian fusion, the fusion posture estimation result of multiple models is:
[0140]
[0141] In general, by training multiple spatiotemporal graph attention network sub-models in divided sliding windows and combining their pose estimation results using a Bayesian fusion integration method based on the NUTS algorithm, we can effectively process time-dependent data and improve the overall performance of the model.
[0142] The present invention proposes a method and system for estimating a person's posture based on multi-model graph neural network fusion. It mainly includes three modules: a human key point extraction module based on a graph autoencoder, a posture recognition module based on a spatiotemporal graph attention network, and a posture estimation fusion module based on multi-model integrated learning. The human key point module based on the graph autoencoder first processes the input image sequence, extracts features at different scales and directions by encoding and reconstructing the graph data, effectively retains the topological structure and spatial relationship of each part of the human body in the image, and then restores the key point position of the human body even in the case of severe occlusion through the dual processing of double-layer graph convolution network structure feature extraction and target classification; the posture recognition module based on the spatiotemporal graph attention network dynamically adjusts the connection weights between different nodes, thereby more accurately capturing the spatial relationship and dependence between various parts of the human body. The spatiotemporal attention mechanism improves the model's processing ability for time series data, and better tracks and adapts to the temporal and spatial changes of the human body; the posture estimation fusion module based on multi-model integrated learning constructs a sliding window to train multiple spatiotemporal graph attention network sub-models, which can effectively process time series data and reduce the risk of overfitting of the model. Then, the posterior weight of each sub-model is obtained through the Bayesian fusion method based on the NUTS algorithm, and the estimation results of multiple sub-models are integrated to reduce the possible deviation and error of a single model, so that the model can show higher stability when processing various complex postures.
[0143] Embodiment 2
[0144] Embodiment 2 of the present disclosure provides a personnel posture estimation system based on multi-model graph neural network fusion, including:
[0145] The data acquisition module is configured to: acquire a human posture image sequence, and divide the acquired data set into a plurality of sub-data sets using a sliding window;
[0146] The human body key point extraction module is configured to: construct the image sequence into a graph structure, convert the graph structure into a low-dimensional graph representation, reconstruct the graph representation, and extract the human body key point sequence based on the reconstructed image;
[0147] The posture recognition module is configured to: construct a skeleton sequence graph according to the key points of the human body, construct a spatiotemporal graph on the skeleton sequence, perform multi-layer graph convolution operations on the spatiotemporal graph to obtain a higher-level feature graph, and determine the posture action category based on the higher-level feature graph;
[0148] The posture estimation fusion module is configured to: fuse the posture estimation results on each sub-dataset to obtain the final posture estimation result.
[0149] The more detailed steps are the same as those in the first embodiment and will not be repeated here.
[0150] Embodiment 3
[0151] Embodiment 3 of the present disclosure provides a medium on which a program is stored. When the program is executed by a processor, the steps in a method for estimating a person's posture based on multi-model graph neural network fusion as described in Embodiment 1 of the present disclosure are implemented.
[0152] The more detailed steps are the same as those in the first embodiment and will not be repeated here.
[0153] Embodiment 4
[0154] Embodiment 4 of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in a method for estimating a person's posture based on multi-model graph neural network fusion as described in Embodiment 1 of the present disclosure are implemented.
[0155] The more detailed steps are the same as those in the first embodiment and will not be repeated here.
[0156] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A personnel posture estimation method based on multi-model graph neural network fusion, characterized in that: The following steps are involved: Collect human posture image sequences, and use sliding windows to divide the collected data sets into multiple sub-data sets; The image sequence is constructed into a graph structure, the graph structure is converted into a low-dimensional graph representation, the graph representation is reconstructed, and the key point sequence of the human body is extracted based on the reconstructed image; A skeleton sequence graph is constructed based on the key points of the human body, and a spatiotemporal graph is constructed on the skeleton sequence. Multi-layer graph convolution operations are performed on the spatiotemporal graph to obtain a higher-level feature graph, and the posture action category is determined based on the higher-level feature graph. The pose estimation results on each sub-dataset are fused to obtain the final pose estimation result.
2. The method for estimating a person's posture based on multi-model graph neural network fusion as described in claim 1, characterized in that: The image sequence is constructed as a graph structure, specifically: Each frame image in the image sequence is divided into multiple parts as graph nodes, the edges of the graph are constructed according to the similarity of each node, and an undirected graph is constructed based on the nodes and edges of the graph.
3. The method for estimating a person's posture based on multi-model graph neural network fusion as described in claim 1, characterized in that: The key point sequence of the human body is extracted based on the reconstructed image, specifically: the reconstructed image is input into the mask prediction network to obtain the mask image sequence, and the position of the joint point is extracted from the obtained mask image sequence.
4. The method for estimating a person's posture based on multi-model graph neural network fusion as described in claim 1, characterized in that: Construct a spatiotemporal graph on the skeleton sequence, specifically: According to the connectivity of the human body structure, the joints in one frame are connected, and then each joint is connected to the same joint in the consecutive frames.
5. The method for estimating a person's posture based on multi-model graph neural network fusion as described in claim 4, characterized in that: The edges of the spatiotemporal graph consist of two subsets, the first subset is the intra-frame edges, connecting different joints in each frame, and the second subset is the inter-frame edges, connecting the same joints in consecutive frames.
6. The method for estimating a person's posture based on multi-model graph neural network fusion as described in claim 1, characterized in that: The graph convolution operation includes spatial graph convolution and temporal graph convolution. The spatial graph convolution adopts a spatial attention mechanism to adaptively capture the dynamic correlation between nodes in the spatial dimension, and the temporal graph convolution adopts a temporal attention mechanism to capture the dynamic information between different frames in an image sequence.
7. The method for estimating a person's posture based on multi-model graph neural network fusion as described in claim 1, characterized in that: The pose estimation results on each sub-dataset are fused to obtain the final pose estimation result, which is as follows: The Bayesian fusion method based on the NUTS algorithm is used to construct the posterior probability distribution of each sub-model, determine the weight of each sub-model, and combine the pose estimation results of multiple sub-models to obtain the final pose estimation.
8. A personnel posture estimation system based on multi-model graph neural network fusion, characterized in that: include: The data acquisition module is configured to: acquire a human posture image sequence, and divide the acquired data set into a plurality of sub-data sets using a sliding window; The human body key point extraction module is configured to: construct the image sequence into a graph structure, convert the graph structure into a low-dimensional graph representation, reconstruct the graph representation, and extract the human body key point sequence based on the reconstructed image; The posture recognition module is configured to: construct a skeleton sequence graph according to the key points of the human body, construct a spatiotemporal graph on the skeleton sequence, perform multi-layer graph convolution operations on the spatiotemporal graph to obtain a higher-level feature graph, and determine the posture action category based on the higher-level feature graph; The posture estimation fusion module is configured to: fuse the posture estimation results on each sub-dataset to obtain the final posture estimation result.
9. A medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the method for personnel posture estimation based on multi-model graph neural network fusion as described in any one of claims 1 to 7 are implemented.
10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the personnel posture estimation method based on multi-model graph neural network fusion as described in any one of claims 1 to 7 are implemented.
Citation Information
Cited By
Walking race video foul action automatic detection method based on 2D human body posture estimation
CN119541049A
Automatic detection method for rule violation actions in race walking video based on 2D human pose estimation
CN119541049B
Pedestrian tracking method and device based on multi-modal data, equipment and storage medium
CN120067746A
Pedestrian tracking method and device based on multi-modal data, equipment and storage medium
CN120067746B
Sparse view angle multi-human body joint reconstruction method based on dynamic graph convolutional network
CN120259572A
