Self-adaptive attitude estimation method based on global feature collaboration and graph structure driving

Abstract

The invention relates to a self-adaptive attitude estimation method based on global feature collaboration and graph structure driving, which comprises the following steps of: S1, performing preliminary feature extraction by adopting a ResNet backbone network to obtain a multi-scale feature graph containing human body global form and joint position detail information; s2, inputting the multi-scale feature map into a global adaptive feature fusion module for adaptive feature fusion, integrating global and local information, and capturing an overall structure and details of a human body posture; s3, inputting the adaptive fusion features obtained in the step S2 into a dynamic visual feature encoder, and optimizing key point identification and positioning through a deformable multi-head attention mechanism; and S4, analyzing the mutual dependency relationship between the key points by adopting a graph reasoning technology, and further refining the position prediction of the key points. The method is beneficial to improving the accuracy of attitude estimation in a complex scene.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an adaptive posture estimation method based on global feature collaboration and graph structure driving. Background Art

[0002] In recent years, with the rapid development of computer vision technology, posture estimation methods based on deep learning have become a research hotspot in this field. These methods have made significant breakthroughs on public data sets and continue to set new performance records. As an important branch of computer vision, posture recognition technology has a wide range of application scenarios. From human-computer interaction to virtual reality, from security monitoring to medical rehabilitation, posture recognition plays a vital role. However, although posture estimation technology has achieved good results in many scenarios, its performance is often greatly challenged when facing complex environments. Complex environments may contain a variety of uncertain factors such as lighting changes, occlusions, background interference, and changing human postures, all of which may have a serious impact on the accuracy of posture estimation. Therefore, how to improve the accuracy of posture estimation in complex scenarios has become a problem that needs to be solved in the current field of posture estimation. Summary of the invention

[0003] The purpose of the present invention is to provide an adaptive posture estimation method based on global feature collaboration and graph structure driving, which is conducive to improving the accuracy of posture estimation in complex scenes.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is: an adaptive posture estimation method based on global feature collaboration and graph structure driving, comprising the following steps:

[0005] Step S1: Use the ResNet backbone network to perform preliminary feature extraction to obtain a multi-scale feature map containing the global morphology of the human body and detailed information of joint positions;

[0006] Step S2: Input the multi-scale feature map into the global adaptive feature fusion module for adaptive feature fusion, integrating global and local information to capture the overall structure and details of human posture;

[0007] Step S3: input the adaptive fusion features obtained in step S2 into the dynamic visual feature encoder, and optimize the key point recognition and positioning through the deformable multi-head attention mechanism;

[0008] Step S4: Use graph reasoning technology to analyze the interdependencies between key points and further refine the key point position prediction.

[0009] Furthermore, the step S1 specifically includes the following steps:

[0010] Step S11: Obtain a public human posture estimation dataset from the Internet, and obtain relevant annotations of the data in the dataset;

[0011] Step S12: ResNet-50 is used as the feature extraction backbone network to preliminarily extract multi-scale feature maps from the input data;

[0012] Step S13: In the absence of differentiable matching, the minimum BatchSize is first set to 14, the initial learning rate α0 is set to 0.001, and the cosine decay strategy is used for adjustment. The network optimization is performed by the Adam optimizer.

[0013] Furthermore, the step S2 specifically includes the following steps:

[0014] Step S21: For the multi-scale feature map extracted in step S1, i Calculate global attention weight and local attention weights Where i represents the level, taking C3, C4, C5; the global attention weight By X i Perform global average pooling (GAP) and pass through a fully connected layer (FC) to obtain; the local attention weight By i Apply a small 3×3 convolution kernel to get it;

[0015]

[0016] Step S22: Get the global attention weight And the local attention weight Weight each layer of features in the multi-scale feature map, and fuse the global and local weighted features to form enhanced features

[0017]

[0018] Among them, σ(·) is the activation function used to normalize the weights;

[0019] Step S23: In order to integrate information of different scales, the enhanced features of the C3, C4, and C5 layers are concatenated in the channel dimension through the Concat(·) function, and then the concatenated features are further optimized through convolution to meet the subsequent task requirements:

[0020]

[0021] Among them, X adaptiverepresents the final adaptive fusion feature, which combines the comprehensive information of global context and local details.

[0022] Furthermore, the step S3 specifically includes the following steps:

[0023] Step S31: Adaptively fusion feature X obtained in step S2 adaptive Input to the encoder for processing; in the encoder, the deformable attention mechanism first processes X through a set of queries Q, keys K, and values ​​V adaptive , where Q, K, and V are obtained by the following formula:

[0024] Q=X adaptive W Q

[0025] K=X adaptive W K

[0026] V=X adaptive W V

[0027] Among them, W Q , W K , W V are the weight matrices for query, key, and value, respectively;

[0028] Step S32: After processing in step S31, the deformable attention mechanism dynamically adjusts the focus position of the key K and value V by learning the offset, so as to focus on the features more flexibly; this process is achieved through the offset function f offsets to implement it, it adjusts K and V according to the query Q:

[0029] K'=K+f offset (Q)

[0030] V'=V+f offset (Q)

[0031] Among them, K' and V' are the adjusted key and value, which can dynamically adjust the focus area according to the background noise;

[0032] Step S33: The encoder uses the adjusted K' and V' and the original query Q to calculate the deformable multi-head attention output to obtain the transformed feature X trans :

[0033]

[0034] Among them, the softmax function ensures the regularization of attention weights, and d k is the dimension of the key vector, used to scale the dot product for training stability.

[0035] Furthermore, the step S4 specifically includes the following steps:

[0036] Step S41: define the human body structure as an undirected graph G = (Ver, E), where Ver is a vertex set representing the joints of the human body, i.e., key points, and E is an edge set representing the physical connection between the key points; use an adjacency matrix to formally describe this human body structure map, where each element of the matrix identifies whether a pair of key points are directly connected; assuming that the human body has key points, then the dimension of the adjacency matrix A is Each element A mn Indicates whether the mth and nth key points are directly connected; if there is a direct physical connection between key points m and n, then A mn =1, otherwise A mn =0;

[0037] Step S42: Combine graph reasoning technology with the decoder to further enhance the ability to detect human joints. In the early stage of the decoder, the features of each node, i.e., key points, are enhanced using predefined human structure map information. This enhancement operation is achieved by aggregating the features of neighboring nodes, expressed as:

[0038]

[0039] Among them, x' id is the enhanced feature, N(id) is the neighbor set of node id, W0 and W1 correspond to the weight matrices of node features and neighbor features respectively, and σ is a nonlinear activation function;

[0040] Step S43: Enhanced node feature x' id Processed through a fully connected layer (FC) to predict the location of key points; expressed as:

[0041] y id =FC(x' id )=σ(W fc x' id +b fc )

[0042] Among them, W fc and b fc are the weights and biases of the fully connected layer, y id is the predicted key point location;

[0043] Step S44: A multi-head attention mechanism is introduced after the fully connected layer; the attention of each head is expressed as:

[0044]

[0045] in, are query, key and value respectively, through the enhanced feature x' id Calculated, d k is the dimension of the key vector; the output of the multi-head attention mechanism is passed through the fully connected layer to refine the predicted key point position to further improve the accuracy of the prediction;

[0046] Finally, the model outputs a prediction of the location of each keypoint;

[0047] Step S45: Use classification loss L c The classification head applied to the pose decoder; in the pose and joint decoders, L1 loss L is used respectively. r and OKS loss L oks To regress pose and joints, we use the heatmap loss L h To assist training to achieve fast convergence; OKS loss L oks The specific formula is as follows:

[0048]

[0049] in, represents the Euclidean distance between the true value and the predicted value; s represents the scale of the target key point, v o A sign indicating whether the true value is visible, δ is the Dirac function, k o represents the key point constant;

[0050] The overall loss function formula is:

[0051] L=L c +αL r +βL h +γL oks

[0052] Among them, α, β, and γ are loss weights.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. The present invention can extract and process feature maps containing key information such as the global shape and joint position of the human body, effectively identify the human body posture in complex environments, improve the accuracy of posture estimation, and effectively reduce the impact of environmental noise.

[0055] 2. The dynamic visual feature encoder and its deformable multi-head attention mechanism enable this method to adapt to complex and changeable human postures and improve the accuracy of key point recognition and positioning.

[0056] 3. The present invention uses graph reasoning technology to enable the present invention to have a deeper understanding of the interdependence between key points, further improve the accuracy and stability of posture estimation, and solve the problem caused by multi-person occlusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flow chart of the method implementation of the embodiment of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0059] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0060] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0061] like Figure 1 As shown, this embodiment provides an adaptive posture estimation method based on global feature collaboration and graph structure driving, including the following steps:

[0062] Step S1: Use the ResNet backbone network to perform preliminary feature extraction to obtain a multi-scale feature map containing the global morphology of the human body and detailed information of joint positions;

[0063] Step S2: Input the multi-scale feature map into the global adaptive feature fusion module for adaptive feature fusion, integrating global and local information to capture the overall structure and details of human posture;

[0064] Step S3: input the adaptive fusion features obtained in step S2 into the dynamic visual feature encoder, and optimize the key point recognition and positioning through the deformable multi-head attention mechanism;

[0065] Step S4: Use graph reasoning technology to analyze the interdependencies between key points and further refine the key point position prediction.

[0066] In this embodiment, the specific implementation steps of step S1 are as follows.

[0067] Step S11: Obtain a public human posture estimation dataset from the Internet and obtain relevant annotations of the data in the dataset.

[0068] Step S12: Using the data obtained from the data set as input, using ResNet-50 as the feature extraction backbone network, and preliminarily extracting a multi-scale feature map of the input data.

[0069] Step S13: In the absence of differentiable matching, the minimum BatchSize is first set to 14, the initial learning rate α0 is set to 0.001, and the cosine decay strategy is used for adjustment, and the network optimization is performed by the Adam optimizer; in this embodiment, the image input size of the CrowdPose data set is 320×256, and the learning rate adjustment of CrowdPose decays to one tenth of the original at the 40th and 70th iterations, respectively.

[0070] In this embodiment, the specific implementation steps of step S2 are as follows.

[0071] Step S21: For the multi-scale feature map extracted in step S1, i (where i represents the level, take C3, C4, C5) Calculate the global attention weight and local attention weights The global attention weight By X i Perform global average pooling (GAP) and pass through a fully connected layer (FC) to obtain; the local attention weight By i Apply a small 3×3 convolution kernel to get it;

[0072]

[0073] Step S22: Get the global attention weight And the local attention weight Weight each layer of features in the multi-scale feature map, and fuse the global and local weighted features to form enhanced features

[0074]

[0075] Here, σ(·) is an activation function, such as Sigmoid, used to normalize the weights, and · represents element-wise multiplication.

[0076] Step S23: In order to integrate information of different scales, the enhanced features of the C3, C4, and C5 layers are concatenated in the channel dimension through the Concat(·) function, and then the concatenated features are further optimized through convolution to meet the subsequent task requirements:

[0077]

[0078] Among them, X adaptive represents the final adaptive fusion feature, which combines the comprehensive information of global context and local details.

[0079] In this embodiment, the specific implementation steps of step S3 are as follows.

[0080] Step S31: Adaptively fusion feature X obtained in step S2 adaptive Input to the encoder for processing. Specifically, suppose X adaptive is the feature matrix obtained from the input data after preprocessing and adaptive feature extraction. In the encoder, the deformable attention mechanism first processes X through a set of queries Q, keys K, and values ​​V. adaptive , where Q, K, and V are obtained by the following formula:

[0081] Q=X adaptive W Q

[0082] K=X adaptive W K

[0083] V=X adaptive W V

[0084] Among them, W Q , W K , W V are the weight matrices for query, key, and value, respectively.

[0085] Step S32: After processing in step S31, the deformable attention mechanism dynamically adjusts the focus position of the key K and value V by learning the offset, so as to focus on the features more flexibly. This process can be achieved by using the offset function f offsets to implement it, it adjusts K and V according to the query Q:

[0086] K'=K+f offset (Q)

[0087] V'=V+f offset (Q)

[0088] Where K' and V' are the adjusted key and value, which can dynamically adjust the focus area according to the background noise.

[0089] Step S33: The encoder uses the adjusted K' and V' and the original query Q to calculate the deformable multi-head attention output to obtain the transformed feature X trans :

[0090]

[0091] Among them, the softmax function ensures the regularization of attention weights, and d k is the dimension of the key vector, used to scale the dot product for training stability.

[0092] In this embodiment, the specific implementation steps of step S4 are as follows.

[0093] Step S41: Define the human body structure as an undirected graph G = (Ver, E), where Ver is a vertex set representing the joints of the human body, i.e., key points, and E is an edge set representing the physical connection between key points. Furthermore, the adjacency matrix is ​​used to formally describe this human body structure map, where each element of the matrix identifies whether a pair of key points is directly connected. Such a mathematical expression not only facilitates the calculation and analysis of the map, but also provides an intuitive data input form for subsequent machine learning models. Assume that the human body has key points, then the dimension of the adjacency matrix A is Each element A mn Indicates whether the mth and nth key points are directly connected; if there is a direct physical connection between key points m and n, then A mn =1, otherwise A mn =0.

[0094] Step S42: Combine graph reasoning technology with the decoder to further enhance the ability to detect human joints; in the early stage of the decoder, use predefined human structure atlas information to enhance the features of each node (key point). This can be achieved by combining the node features with the aggregated features of its neighbors, thereby introducing graph structure information without changing the overall network architecture. This process not only includes the mathematical expression of the human structure atlas, but also involves specific steps on how to optimize the posture estimation process through this atlas. Specifically, the atlas information is first used to enhance the features of each node (i.e., key point); this enhancement operation is achieved by aggregating the features of neighboring nodes, expressed as:

[0095]

[0096] Among them, x' id is the enhanced feature, N(id) is the neighbor set of node id, W0 and W1 correspond to the weight matrices of node features and neighbor features respectively, and σ is a nonlinear activation function.

[0097] Step S43: Enhanced node feature x' id It is processed through a fully connected layer (FC) to predict the location of the key points. The role of the fully connected layer is to transform the input features so as to learn the nonlinear relationship from the feature to the key point location mapping. This can be expressed as:

[0098] y id =FC(x′ id )=σ(W fc x id +b fc )

[0099] Among them, W fc and b fc are the weights and biases of the fully connected layer, y id is the predicted key point location.

[0100] Step S44: Introduce a multi-head attention mechanism after the fully connected layer. Here, the multi-head attention mechanism can help the model capture the dependencies between different key points, especially in complex human postures. At this stage, the attention of each head can be expressed as:

[0101]

[0102] in, are query, key and value respectively, through the enhanced feature x' id Calculated, d k is the dimension of the key vector; the output of the multi-head attention mechanism is passed through the fully connected layer to refine the predicted key point position, which can further improve the accuracy of the prediction.

[0103] Finally, the model outputs a prediction of the location of each keypoint;

[0104] Step S45: Use classification loss (denoted as L c ), applied to the classification head of the pose decoder. In the pose and joint decoders, L1 loss (denoted as L r ) and OKS loss (denoted as L oks ) to regress the pose and joints. At the same time, the heat map loss (denoted as L h ) is used to assist training to achieve fast convergence, but it is only used in training and not in reasoning. When dealing with postures with large scale differences, the L1 loss shows limitations due to the situation where the relative errors are similar; therefore, this method supplements the use of the object keypoint similarity (OKS) loss to improve this problem. The OKS loss L oks The specific formula is as follows:

[0105]

[0106] in, represents the Euclidean distance between the true value and the predicted value; s represents the scale of the target key point, v o A sign indicating whether the true value is visible, δ is the Dirac delta function, k o represents the key point constant;

[0107] The overall loss function formula is:

[0108] L=L c +αL r +βL h +γL oks

[0109] Among them, α, β, and γ are loss weights.

[0110] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0111] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0114] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. An adaptive pose estimation method based on global feature collaboration and graph structure driving, It is characterized in that The following steps are involved: Step S1: Use the ResNet backbone network to perform preliminary feature extraction to obtain a multi-scale feature map containing the global morphology of the human body and detailed information of joint positions; Step S2: Input the multi-scale feature map into the global adaptive feature fusion module for adaptive feature fusion, integrating global and local information to capture the overall structure and details of human posture; Step S3: input the adaptive fusion features obtained in step S2 into the dynamic visual feature encoder, and optimize the key point recognition and positioning through the deformable multi-head attention mechanism; Step S4: Use graph reasoning technology to analyze the interdependencies between key points and further refine the key point position prediction.

2. The adaptive posture estimation method based on global feature collaboration and graph structure driving according to claim 1, It is characterized in that The step S1 specifically includes the following steps: Step S11: Obtain a public human posture estimation dataset from the Internet, and obtain relevant annotations of the data in the dataset; Step S12: ResNet-50 is used as the feature extraction backbone network to preliminarily extract multi-scale feature maps from the input data; Step S13: In the absence of differentiable matching, the minimum BatchSize is set to 14 and the initial learning rate α 0 It is set to 0.001 and adjusted using the cosine decay strategy, and the network optimization is performed using the Adam optimizer.

3. The adaptive posture estimation method based on global feature collaboration and graph structure driving according to claim 2, It is characterized in that The step S2 specifically includes the following steps: Step S21: For the multi-scale feature map extracted in step S1, i Calculate global attention weight and local attention weights Where i represents the level, taking C3, C4, C5; the global attention weight By X i Perform global average pooling (GAP) and pass through a fully connected layer (FC) to obtain; the local attention weight By i Apply a small 3×3 convolution kernel to get it; Step S22: Get the global attention weight And the local attention weight Weight each layer of features in the multi-scale feature map, and fuse the global and local weighted features to form enhanced features Among them, σ(·) is the activation function used to normalize the weights; Step S23: In order to integrate information of different scales, the enhanced features of the C3, C4, and C5 layers are concatenated in the channel dimension through the Concat(·) function, and then the concatenated features are further optimized through convolution to meet the subsequent task requirements: Among them, X adaptive represents the final adaptive fusion feature, which combines the comprehensive information of global context and local details.

4. The adaptive posture estimation method based on global feature collaboration and graph structure driving according to claim 3, It is characterized in that The step S3 specifically comprises the following steps: Step S31: Adaptively fusion feature X obtained in step S2 adaptive Input to the encoder for processing; in the encoder, the deformable attention mechanism first processes X through a set of queries Q, keys K, and values ​​V adaptive , where Q, K, and V are obtained by the following formula: Q=X adaptive W Q K=X adaptive W K V=X adaptive W V Among them, W Q , W K , W V are the weight matrices for query, key, and value, respectively; Step S32: After processing in step S31, the deformable attention mechanism dynamically adjusts the focus position of the key K and value V by learning the offset, so as to focus on the features more flexibly; this process is achieved through the offset function f offsets to implement it, it adjusts K and V according to the query Q: K’=K+f offset (Q) V′=V+f offset (Q) Among them, K′ and V′ are the adjusted key and value, which can dynamically adjust the focus area according to the background noise; Step S33: The encoder uses the adjusted K′ and V′ and the original query Q to calculate the deformable multi-head attention output to obtain the transformed feature X trans : Among them, the softmax function ensures the regularization of attention weights, and d k is the dimension of the key vector, used to scale the dot product for training stability.

5. The method for adaptive posture estimation based on global feature collaboration and graph structure driving according to claim 4, It is characterized in that The step S4 specifically comprises the following steps: Step S41: define the human body structure as an undirected graph G = (Ver, E), where Ver is a vertex set representing the joints of the human body, i.e., key points, and E is an edge set representing the physical connection between the key points; use an adjacency matrix to formally describe this human body structure map, where each element of the matrix identifies whether a pair of key points are directly connected; assuming that the human body has key points, then the dimension of the adjacency matrix A is Each element A mn Indicates whether the mth and nth key points are directly connected; if there is a direct physical connection between key points m and n, then A mn =1, otherwise A mn =0; Step S42: Combine graph reasoning technology with the decoder to further enhance the ability to detect human joints. In the early stage of the decoder, the features of each node, i.e., key points, are enhanced using predefined human structure map information. This enhancement operation is achieved by aggregating the features of neighboring nodes, expressed as: Among them, x′ id is the enhanced feature, N(id) is the neighbor set of node id, W 0 and W 1 The weight matrices corresponding to the node's own features and neighbor features respectively, σ is a nonlinear activation function; Step S43: Enhanced node feature x′ id Processed through a fully connected layer (FC) to predict the location of key points; expressed as: y id =FC(x′ id )=σ(W fc x′ id +b fc ) Among them, W fc and b fc are the weights and biases of the fully connected layer, y id is the predicted key point location; Step S44: A multi-head attention mechanism is introduced after the fully connected layer; the attention of each head is expressed as: in, are query, key and value respectively, through the enhanced feature x′ id Calculated, d k is the dimension of the key vector; the output of the multi-head attention mechanism is passed through the fully connected layer to refine the predicted key point position to further improve the accuracy of the prediction; Finally, the model outputs a prediction of the location of each keypoint; Step S45: Use classification loss L c The classification head applied to the pose decoder; in the pose and joint decoders, L1 loss L is used respectively. r and OKS loss L oks To regress pose and joints, we use the heatmap loss L h To assist training to achieve fast convergence; OKS loss L oks The specific formula is as follows: in, represents the Euclidean distance between the true value and the predicted value; s represents the scale of the target key point, v o A sign indicating whether the true value is visible, δ is the Dirac function, k o represents the key point constant; The overall loss function formula is: L=L c +αL r +βL h +γL oks Among them, α, β, γ are loss weights.

Images