3D human body posture estimation method and system based on angle diagram and image feature fusion

Abstract

The invention discloses a 3D human body posture estimation method and system based on angle diagram and image feature fusion, and the method comprises the steps: obtaining a 2D key point coordinate of a human body through employing a trained 2D human body posture detector, and obtaining an original image with obvious edge features through employing a Gaussian Laplacian operator; converting the topological graph of the human skeleton into two different angle diagrams; an angle diagram feature comparison and fusion module is constructed, and angle diagram features of different scales are fused through cross comparison learning for cross comparison; the image features and the angle graph features are fused, and finally regression is carried out to obtain 3D human body posture information. And repeating training to obtain a final 3D human body posture estimation model. According to the method, the joint angle constraint is introduced into the graph network, the influence caused by depth blur is reduced, and meanwhile, the image features are fused, so that the network has better expression ability.

Description

Technical Field

[0001] The present invention belongs to the field of 3D human body posture estimation, and in particular relates to a 3D human body posture estimation method and system based on angle map and image feature fusion. Background Art

[0002] The existing 3D human pose estimation methods can be mainly divided into one-stage methods and two-stage methods. The one-stage method does not require a 2D estimator to directly regress the 3D pose. In contrast, the two-stage method is based on the 2D joint point position and connects the 2D and 3D estimators in series. Thanks to the outstanding work in the field of 2D pose estimation in recent years, more and more researchers have devoted their energy to the two-stage method. The key to the two-stage method is the dimensionality improvement from 2D to 3D. However, due to the inherent depth ambiguity problem of 2D pose regression to 3D pose, that is, the same 2D pose can be obtained from multiple different 3D poses, the regression accuracy of the two-stage method still has a lot of room for improvement.

[0003] In order to reduce the accuracy impact caused by depth blur, researchers have made many attempts. For example, the invention patent with patent number CN202110546337.2 established a back-projection network, which implemented motion constraints on different joints by grouping joint points, thereby reducing depth blur to a certain extent; for example, the invention patent with patent number CN202210142459.X also used a similar grouping idea to capture the local and global features of human posture. However, this type of method is limited in its effect on overcoming the problem of depth blur.

[0004] There are also methods that focus on the training data level. For example, the patent with patent number CN202210460455.6 proposes a multi-view feature fusion method to obtain more robust depth features by extracting 2D posture expressions from multiple views. However, this type of method faces two problems: most existing data sets only provide single-view images, and there is very little multi-view data for training, so its generalization is questionable in actual application scenarios; at the application level, providing information from multiple perspectives often means greater costs. Therefore, the patent with patent number CN202110569187.7 invented a method based on joint data enhancement and network training model, attempting to improve the quality of training data through data enhancement, but it still does not essentially solve the problem of depth ambiguity well. Summary of the invention

[0005] The present invention aims to overcome the above-mentioned shortcomings of the prior art and proposes a 3D human posture estimation method and system based on angle map and image feature fusion to provide support for solving the problem of human posture estimation from 2D to 3D.

[0006] To achieve the above object, the present invention provides the following scheme: a 3D human body posture estimation method based on angle map and image feature fusion, comprising the following steps:

[0007] S1: Construct the original data set, use the trained 2D human pose detector to obtain the 2D key point coordinates of the human body, and use the Gaussian Laplacian operator to obtain the original image with obvious edge features;

[0008] S2: Convert the original skeleton graph G with human joints as nodes into a line graph with bones as nodes;

[0009] S3: Convert the original skeleton graph G into a second-order graph that explicitly represents the angle relationship between joints;

[0010] S4: Construct an angle graph feature comparison and fusion module to fuse angle graph features of different scales through cross-comparison learning; the angle graph feature comparison and fusion module consists of a common graph neural network layer and an unpooling graph convolution layer. The former is used to extract the features of line graphs and second-order graphs, and the latter uses unpooling operations to keep the dimensions of the intermediate features of line graphs consistent with those of the intermediate features of second-order graphs. Finally, feature fusion is completed through learnable weight parameters.

[0011] S5: Train the angle map feature comparison and fusion module to obtain the intermediate model: input the processed image into the convolutional neural network model to obtain image features, and fuse the image features with the angle map features, and finally regress to obtain 3D human posture information;

[0012] S6: Perform supervised fine-tuning on the intermediate model: adjust the parameters of the regression head of the intermediate model, freeze other parameters of the intermediate model, and obtain the final 3D human pose estimation model.

[0013] Preferably, step S1 specifically includes:

[0014] S1.1: Transform the two-dimensional images in the Human3.6M dataset The coordinates of 16 2D key points are obtained after normalization through the cascade pyramid network. ,in ;

[0015] S1.2: Using the Laplacian of Gaussian Edge sharpening is performed , whose resolution is ;

[0016] The Laplace operator of Gaussian is:

[0017] (1)

[0018] in, represents the standard deviation, , Represent the coordinates of the pixel points, Represents the natural logarithm.

[0019] Preferably, step S2 specifically includes:

[0020] S2.1: The original skeleton graph with human joints as nodes Converted to a line graph with human bones as nodes ;

[0021] in Represented as the original skeleton graph and line graph The node set and edge set of and , Line graph The nodes are the edges of the original skeleton graph, expressed as , and the edge is represented by ;Two edges in the original skeleton graph are connected if and only if they have a common node;

[0022] S2.2: Calculate the original skeleton graph The angle between two connected sides in the graph is plotted as a line The edge weights of the original skeleton graph are taken as the coordinates of the two end points of the edge. New input features ;

[0023] in , the line graph The adjacency matrix of ,in , It can be obtained by the following formula:

[0024] (2)

[0025] in is a binary matrix, representing the node set in the original graph Is the node of the edge set The endpoints of the middle edge, is represented as the identity matrix, and F(·) is represented as the function that calculates the joint angles in the original skeleton graph.

[0026] Preferably, the step S3 specifically includes:

[0027] S3.1: The original skeleton graph Converted into a second-order graph that directly represents the angle relationship between joint points ;

[0028] in , represents the edges between nodes with a path length of 2 in the original skeleton graph.

[0029] S3.2: Compute the second-order graph The adjacency matrix of Regenerate node features ;

[0030] The original skeleton graph The adjacency matrix of ,in , , a value of 1 indicates that the two nodes are connected, and a value of 0 indicates that they are not connected; a second-order graph representing the angle The adjacency matrix of It is expressed as:

[0031] (3)

[0032] Second Order Graph Node features ,when and When is a second-order neighbor, and The angle value between or and When it is not a second-order neighbor, .

[0033] Preferably, the step S4 comprises:

[0034] S4.1: Use ordinary graph neural network layers to classify line graphs separately and the second-order graph Perform feature extraction and generate intermediate features and ,in is the feature dimension, and the original image Apply ordinary graph neural network to generate intermediate features ;

[0035] The general graph neural network layer is represented as:

[0036] (4)

[0037] The above formula is further described as:

[0038] (5)

[0039] (6)

[0040] in, represents the learnable weight matrix, Line graph The adjacency matrix of represents the activation function, represents the modulation matrix, represents the Hadamard product, Represented by the second-order graph Or original image The adjacency matrix obtained after symmetric normalization of the adjacency matrix;

[0041] S4.2: The intermediate features of the line graph are transformed into dimensions through the unpooling graph neural network. .

[0042] The depooling graph neural network layer includes two separate graph convolutional networks, which are used to obtain the embedding matrix and the alignment matrix respectively:

[0043] (7)

[0044] (8)

[0045] in, represents the embedding matrix, represents the alignment matrix, and finally, the line graph The second-order graph is obtained by the following formula Middle layer features with consistent dimensions:

[0046] (9)

[0047] in, Represents the line graph after the anti-pooling operation intermediate features.

[0048] S4.3: Fusion of multi-scale angle map features through cross-contrast learning. Specifically, feature learning is constrained by three different forms of loss functions.

[0049] Cross-compare original image features With line graph features , which is used to enhance the expression of local angle information and edge geometric characteristics. The calculation formula is:

[0050] (10)

[0051] in, Expressed as cosine similarity, Expressed as a temperature parameter, , Represented as vectors of nodes in the original graph and line graph respectively.

[0052] Cross-compare original image features and second-order graph features , which is used to enhance the capture of high-order angle relationships and global structural information. The calculation formula is:

[0053] (11)

[0054] in, Represented as a node Eigenvectors in the second-order graph.

[0055] Cross-comparison of original graph features in attribute module The original graph features in the structure module , used to ensure the consistency of multi-view features, the calculation formula is:

[0056] (12)

[0057] Through the joint constraints of the above contrast loss, the learnable weight parameters are used , fuse the features of the three views to generate intermediate fusion features , the fusion formula is:

[0058] (13)

[0059] in, Represented as a concatenation operation along the feature dimension.

[0060] Preferably, step S5 comprises:

[0061] S5.1: Train the angle graph feature comparison and fusion module to obtain the intermediate model and optimize the loss function by gradient descent , the overall contrast loss of the angle map feature contrast fusion module is:

[0062] (14)

[0063] in, and Represented as a hyperparameter, it is used to balance the loss terms.

[0064] S5.2: Input the processed image into a convolutional neural network model to obtain image features, and fuse the image features with the angle map features. The convolutional neural network model is HRNet, and the image features of different scales in four stages are fused with the angle map features through a conversion module, wherein the conversion module uses a fully connected layer to reduce the two-dimensional image features to one dimension.

[0065] Preferably, the intermediate model supervised fine-tuning process described in step S6 freezes other parameters of the intermediate model, and fine-tunes the output head of the prediction module using a weighted loss function, which is specifically expressed as:

[0066] (15)

[0067] in, Represents the weighted sum of the mean square error and the mean absolute error, and the weighting coefficient .

[0068] The present invention also provides a system for implementing the 3D human body posture estimation method based on angle map and image feature fusion of the present invention, comprising:

[0069] Data preprocessing module, graph network construction module, angle graph feature comparison and fusion module, image feature fusion module, and reasoning module;

[0070] The data preprocessing module obtains the coordinates of 2D key points of the human body from the original image and performs normalization processing, and uses the Gaussian Laplace operator to enhance the edge features of the original image;

[0071] The graph network construction module uses known 2D posture information to construct a network in two different ways, so that the new graph network structure has the ability to represent joint angle information;

[0072] The angle feature comparison and fusion module cross-compares the multi-scale angle image features so that the angle image features of different scales have consistent dimensions and perform feature fusion;

[0073] The image feature fusion module uses a fully connected layer network to fuse two different features, image and graph, at different stages of the model;

[0074] The reasoning module finally outputs the reasoning result of the model, that is, the 3D posture information of the human body;

[0075] The data preprocessing module, graph network construction module, angle graph feature comparison and fusion module, image feature fusion module and reasoning module are connected in sequence.

[0076] The third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the 3D human body posture estimation method based on the fusion of angle graph and image features of the present invention.

[0077] The technical concept of the present invention is:

[0078] After obtaining the original image, the trained 2D posture estimator is first used to detect the 2D joint coordinates of the human body, and the Gaussian Laplacian operator is used for image processing to obtain an image with obvious edge features; secondly, different angle graphs are constructed using the known 2D joint coordinates, and the joint angle information in the human skeleton topological structure is fully utilized to overcome the depth ambiguity problem, and by cross-comparing the multi-scale angle graph features, the angle graph features of different scales have consistent dimensions and feature fusion is performed; in addition, in order to ensure that the graph network does not lose the original information, the convolutional neural network model is used to extract image features and fuse them with the angle graph features to improve the regression accuracy of the final 3D posture.

[0079] The beneficial effects of the present invention are:

[0080] 1) The human skeleton structure itself has topological properties, and graph networks can efficiently represent the features of human joints and the edges between joints;

[0081] 2) Two different angle graph composition methods are proposed, and angle constraints are introduced into the graph network, thereby alleviating the accuracy loss caused by depth blur;

[0082] 3) Using graph neural networks and convolutional neural networks in the same model, using graph neural networks to extract human bone structure features while using convolutional neural networks to extract its visual features;

[0083] 4) The backbone network used to extract image features in the present invention is not specific, and any convolutional network with multi-stage model characteristics can be used in this method, so this method is scalable. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 It is a system framework diagram of the method of the present invention;

[0085] Figure 2 It is a model structure diagram of the method of the present invention;

[0086] Fig. 3 (a) to Fig. 3 (f) are two schematic diagrams of local and global structures describing angle relationships of the present invention, wherein Fig. 3 (a) is a schematic diagram of the angle at the center of the pelvis of the original skeleton graph, Fig. 3 (b) is a schematic diagram of the conversion of the center of the pelvis of the original skeleton graph into a second-order graph, wherein the graph no longer retains the edges between first-order neighbors (indicated by dotted lines) and generates new edges for second-order neighbors (indicated by solid lines), Fig. 3 (c) is a mapping relationship diagram of each connecting edge at the center of the pelvis of the original skeleton graph to the line graph structure, Fig. 3 (d) is a schematic diagram of the original skeleton, Fig. 3 (e) is a schematic diagram of the structure when the original skeleton graph is reconstructed into a second-order graph, and Fig. 3 (f) is a schematic diagram of the structure when the original skeleton graph is reconstructed into a line graph;

[0087] Figure 4 (a) to Figure 4 (b) are schematic diagrams of the angle feature fusion model of the present invention, wherein Figure 4 (a) is an angle feature extraction module, and Figure 4 (b) is a line graph intermediate feature alignment module. DETAILED DESCRIPTION

[0088] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0089] Example 1

[0090] Reference Figure 1 ~ Figure 4 (b), this embodiment relates to a 3D human posture estimation method based on angle map and image feature fusion, the steps are as follows:

[0091] S1: Construct the original data set, use the trained 2D human pose detector to obtain the 2D key point coordinates of the human body, and use the Gaussian Laplacian operator to obtain the original image with obvious edge features;

[0092] S2: As shown in Figure 3 (f), the original skeleton graph with human joints as nodes Convert to a line graph with bones as nodes;

[0093] S3: As shown in Figure 3(e), the original skeleton image Converted into a second-order graph that explicitly represents the angular relationship between joints;

[0094] S4: Construct an angle graph feature comparison and fusion module to fuse angle graph features of different scales through cross-comparison learning; the angle graph feature comparison and fusion module consists of a common graph neural network layer and an unpooling graph convolution layer. The former is used to extract the features of line graphs and second-order graphs, and the latter uses unpooling operations to keep the dimensions of the intermediate features of line graphs consistent with those of the intermediate features of second-order graphs. Finally, feature fusion is completed through learnable weight parameters.

[0095] S5: Train the angle map feature comparison and fusion module to obtain the intermediate model: input the processed image into the convolutional neural network model to obtain image features, and fuse the image features with the angle map features, and finally regress to obtain 3D human posture information;

[0096] S6: Perform supervised fine-tuning on the intermediate model: adjust the parameters of the regression head of the intermediate model, freeze other parameters of the intermediate model, and obtain the final 3D human pose estimation model.

[0097] Step S1 specifically includes:

[0098] S1.1: As shown in Figure 3(d), the two-dimensional image in the Human3.6M dataset The coordinates of 16 2D key points are obtained after normalization through the cascade pyramid network. ,in ;

[0099] S1.2: Using the Laplacian of Gaussian Edge sharpening is performed , whose resolution is ;

[0100] The Laplace operator of Gaussian is:

[0101] (1)

[0102] in, represents the standard deviation, , Represent the coordinates of the pixel points, Represents the natural logarithm.

[0103] Step S2 specifically includes:

[0104] S2.1: The original skeleton graph with human joints as nodes Converted to a line graph with human bones as nodes ;

[0105] in Represented as the original skeleton graph and line graph The node set and edge set of and , as shown in Figure 3(c), the line graph The nodes are the edges of the original skeleton graph, expressed as , and the edge is represented by ;Two edges in the original skeleton graph are connected if and only if they have a common node;

[0106] S2.2: Calculate the original skeleton graph The angle between two connected sides in the graph is plotted as a line The edge weights of the original skeleton graph are taken as the coordinates of the two end points of the edge. Node features ;

[0107] in , the line graph The adjacency matrix of ,in , It can be obtained by the following formula:

[0108] (2)

[0109] in is a binary matrix, representing the node set in the original graph Is the node of the edge set The endpoints of the middle edge, is represented as the identity matrix, and F(·) is represented as the function that calculates the joint angles in the original skeleton graph.

[0110] Step S3 specifically includes:

[0111] S3.1: As shown in Figure 3(b), the original skeleton graph Converted into a second-order graph that directly represents the angle relationship between joint points ;

[0112] in , represents the edges between nodes with a path length of 2 in the original skeleton graph.

[0113] S3.2: Compute the second-order graph The adjacency matrix of Generate node features ;

[0114] The original skeleton graph The adjacency matrix of ,in , , a value of 1 indicates that the two nodes are connected, and a value of 0 indicates that they are not connected; a second-order graph representing the angle The adjacency matrix of It is expressed as:

[0115] (3)

[0116] As shown in Figure 3(b), the second-order graph Node features ,when and When is a second-order neighbor, and The angle value between or and When it is not a second-order neighbor, .

[0117] Step S4 specifically includes:

[0118] S4.1: Use ordinary graph neural network layers to classify line graphs separately and the second-order graph Perform feature extraction and generate intermediate features and ,in is the feature dimension, and the original image Apply ordinary graph neural network to generate intermediate features .

[0119] The general graph neural network layer is represented as:

[0120] (4)

[0121] The above formula is further described as:

[0122] (5)

[0123] (6)

[0124] in, represents the learnable weight matrix, Line graph The adjacency matrix of represents the activation function, represents the modulation matrix, represents the Hadamard product, Represented by the second-order graph Or original image The adjacency matrix obtained after symmetric normalization of the adjacency matrix;

[0125] S4.2: The intermediate features of the line graph are transformed into dimensions through the unpooling graph neural network. .

[0126] The depooling graph neural network layer includes two separate graph convolutional networks, which are used to obtain the embedding matrix and the alignment matrix respectively:

[0127] (7)

[0128] (8)

[0129] in, represents the embedding matrix, represents the alignment matrix, and finally, the line graph The second-order graph is obtained by the following formula Middle layer features with consistent dimensions:

[0130] (9)

[0131] in, Represents the line graph after the anti-pooling operation intermediate features.

[0132] S4.3: Fusion of multi-scale angle map features through cross-contrast learning. Specifically, feature learning is constrained by three different forms of loss functions.

[0133] Cross-compare original image features With line graph features , which is used to enhance the expression of local angle information and edge geometric characteristics. The calculation formula is:

[0134] (10)

[0135] in, Expressed as cosine similarity, Expressed as a temperature parameter, , Represented as vectors of nodes in the original graph and line graph respectively.

[0136] Cross-compare original image features and second-order graph features , which is used to enhance the capture of high-order angle relationships and global structural information. The calculation formula is:

[0137] (11)

[0138] in, Represented as a node Eigenvectors in the second-order graph.

[0139] Cross-comparison of original graph features in attribute module The original graph features in the structure module , used to ensure the consistency of multi-view features, the calculation formula is:

[0140] (12)

[0141] Through the joint constraints of the above contrast loss, the learnable weight parameters are used , fuse the features of the three views to generate intermediate fusion features , the fusion formula is:

[0142] (13)

[0143] in, Represented as a concatenation operation along the feature dimension.

[0144] Step S5 specifically includes:

[0145] S5.1: Train the angle graph feature comparison and fusion module to obtain the intermediate model and optimize the loss function by gradient descent , the overall contrast loss of the angle map feature contrast fusion module is:

[0146] (14)

[0147] in, and Represented as a hyperparameter, it is used to balance the loss terms.

[0148] S5.2: Input the processed image into a convolutional neural network model to obtain image features, and fuse the image features with the angle map features. The convolutional neural network model is HRNet, and the image features of different scales in four stages are fused with the angle map features through a conversion module, wherein the conversion module uses a fully connected layer to reduce the two-dimensional image features to one dimension.

[0149] The intermediate model supervised fine-tuning process in step S6 freezes other parameters of the intermediate model and fine-tunes the output head of the prediction module using a weighted loss function. The specific formula is expressed as:

[0150] (15)

[0151] in, Represents the weighted sum of the mean square error and the mean absolute error, and the weighting coefficient .

[0152] The 3D human posture estimation system based on angle graph and image feature fusion of the present invention includes a data preprocessing module, a graph network construction module, an angle graph feature comparison and fusion module, an image feature fusion module, and an inference module;

[0153] The data preprocessing module obtains the coordinates of the 2D key points of the human body from the original image and performs normalization processing. At the same time, the Gaussian Laplacian operator is used to enhance the edge features of the original image, including:

[0154] S1.1: Transform the two-dimensional images in the Human3.6M dataset The coordinates of 16 2D key points are obtained after normalization through the cascade pyramid network. ,in ;

[0155] S1.2: Using the Laplacian of Gaussian Edge sharpening is performed , whose resolution is ;

[0156] The Laplace operator of Gaussian is:

[0157] (1)

[0158] in, represents the standard deviation, , Represent the coordinates of the pixel points, Represents the natural logarithm.

[0159] The graph network construction module uses the known 2D posture information to construct the network in two different ways, so that the new graph network structure has the ability to represent joint angle information, including:

[0160] S2.1: The original skeleton graph with human joints as nodes Converted to a line graph with human bones as nodes ;

[0161] in Represented as the original skeleton graph and line graph The node set and edge set of and , Line graph The nodes are the edges of the original skeleton graph, expressed as , and the edge is represented by ;Two edges in the original skeleton graph are connected if and only if they have a common node;

[0162] S2.2: Calculate the original skeleton graph The angle between two connected sides in the graph is plotted as a line The edge weights of the original skeleton graph are taken as the coordinates of the two end points of the edge. Node features ;

[0163] in , the line graph The adjacency matrix of ,in , It can be obtained by the following formula:

[0164] (2)

[0165] in is a binary matrix, representing the node set in the original graph Is the node of the edge set The endpoints of the middle edge, is represented as the identity matrix, and F(·) is represented as the function that calculates the joint angles in the original skeleton graph.

[0166] S3.1: The original skeleton graph Converted into a second-order graph that directly represents the angle relationship between joint points ;

[0167] in , represents the edges between nodes with a path length of 2 in the original skeleton graph.

[0168] S3.2: Compute the second-order graph The adjacency matrix of Generate node features ;

[0169] The original skeleton graph The adjacency matrix of ,in , , a value of 1 indicates that the two nodes are connected, and a value of 0 indicates that they are not connected; a second-order graph representing the angle The adjacency matrix of It is expressed as:

[0170] (3)

[0171] Second Order Graph Node features ,when and When is a second-order neighbor, and The angle value between or and When it is not a second-order neighbor, .

[0172] The angle image feature comparison and fusion module cross-compares the multi-scale angle image features to make the angle image features of different scales have consistent dimensions and perform feature fusion. Specifically, it includes:

[0173] S4.1: Use ordinary graph neural network layers to classify line graphs separately and the second-order graph Perform feature extraction and generate intermediate features and ,in is the feature dimension, and the original image Apply ordinary graph neural network to generate intermediate features ;

[0174] The general graph neural network layer is represented as:

[0175] (4)

[0176] The above formula is further described as:

[0177] (5)

[0178] (6)

[0179] in, represents the learnable weight matrix, Line graph The adjacency matrix of represents the activation function, represents the modulation matrix, represents the Hadamard product, Represented by the second-order graph Or original image The adjacency matrix obtained after symmetric normalization of the adjacency matrix;

[0180] S4.2: The intermediate features of the line graph are transformed into dimensions through the unpooling graph neural network. .

[0181] The depooling graph neural network layer includes two separate graph convolutional networks, which are used to obtain the embedding matrix and the alignment matrix respectively:

[0182] (7)

[0183] (8)

[0184] in, represents the embedding matrix, represents the alignment matrix, and finally, the line graph The second-order graph is obtained by the following formula Middle layer features with consistent dimensions:

[0185] (9)

[0186] in, Represents the line graph after the anti-pooling operation intermediate features.

[0187] S4.3: Fusion of multi-scale angle map features through cross-contrast learning. Specifically, feature learning is constrained by three different forms of loss functions.

[0188] Cross-compare original image features With line graph features , which is used to enhance the expression of local angle information and edge geometric characteristics. The calculation formula is:

[0189] (10)

[0190] in, Expressed as cosine similarity, Expressed as a temperature parameter, , Represented as vectors of nodes in the original graph and line graph respectively.

[0191] Cross-compare original image features and second-order graph features , which is used to enhance the capture of high-order angle relationships and global structural information. The calculation formula is:

[0192] (11)

[0193] in, Represented as a node Eigenvectors in the second-order graph.

[0194] Cross-comparison of original graph features in attribute module The original graph features in the structure module , used to ensure the consistency of multi-view features, the calculation formula is:

[0195] (12)

[0196] Through the joint constraints of the above contrast loss, the learnable weight parameters are used , fuse the features of the three views to generate intermediate fusion features , the fusion formula is:

[0197] (13)

[0198] in, Represented as a concatenation operation along the feature dimension.

[0199] Train the angle graph feature comparison and fusion module to obtain the intermediate model, and use the image feature fusion module to use the fully connected layer network to fuse the two different features of image and graph at different stages of the model, including:

[0200] S5.1: Train the angle graph feature comparison and fusion module to obtain the intermediate model and optimize the loss function by gradient descent , the overall contrast loss of the angle map feature contrast fusion module is:

[0201] (14)

[0202] in, and Represented as a hyperparameter, it is used to balance the loss terms.

[0203] S5.2: Input the processed image into a convolutional neural network model to obtain image features, and fuse the image features with the angle map features. The convolutional neural network model is HRNet, and the image features of different scales in four stages are fused with the angle map features through a conversion module, wherein the conversion module uses a fully connected layer to reduce the two-dimensional image features to one dimension.

[0204] After supervised fine-tuning of the intermediate model, the reasoning module finally outputs the reasoning result of the model, that is, the 3D posture information of the human body.

[0205] Example 2

[0206] This embodiment relates to a system for implementing a 3D human body posture estimation method based on angle map and image feature fusion of embodiment 1, including:

[0207] Data preprocessing module, graph network construction module, angle graph feature comparison and fusion module, image feature fusion module, and reasoning module;

[0208] The data preprocessing module obtains the coordinates of 2D key points of the human body from the original image and performs normalization processing, and uses the Gaussian Laplace operator to enhance the edge features of the original image;

[0209] The graph network construction module uses known 2D posture information to construct a network in two different ways, so that the new graph network structure has the ability to represent joint angle information;

[0210] The angle feature comparison and fusion module uses an embedding matrix and an alignment matrix to make the angle image features of different scales have consistent dimensions and perform feature fusion;

[0211] The image feature fusion module uses a fully connected layer network to fuse two different features, image and graph, at different stages of the model;

[0212] The reasoning module finally outputs the reasoning result of the model, that is, the 3D posture information of the human body;

[0213] The data preprocessing module, graph network construction module, angle graph feature comparison and fusion module, image feature fusion module and reasoning module are connected in sequence.

[0214] Example 3

[0215] This embodiment relates to a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the 3D human body posture estimation method based on angle map and image feature fusion of embodiment 1 is implemented.

[0216] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to the equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A 3D human posture estimation method based on angle map and image feature fusion, characterized by: The following steps are involved: S1: Construct the original data set, use the trained 2D human pose detector to obtain the 2D key point coordinates of the human body, and use the Gaussian Laplacian operator to obtain the original image with obvious edge features; S2: The original skeleton graph with human joints as nodes Convert to a line graph with bones as nodes; S3: The original skeleton graph Converted into a second-order graph that explicitly represents the angular relationship between joints; S4: Construct an angle graph feature comparison and fusion module to fuse angle graph features of different scales through cross-comparison learning; the angle graph feature comparison and fusion module consists of a common graph neural network layer and an unpooling graph convolution layer. The former is used to extract the features of line graphs and second-order graphs, and the latter uses unpooling operations to keep the dimensions of the intermediate features of line graphs consistent with those of the intermediate features of second-order graphs. Finally, feature fusion is completed through learnable weight parameters. S5: Train the angle map feature comparison and fusion module to obtain the intermediate model: input the processed image into the convolutional neural network model to obtain image features, and fuse the image features with the angle map features, and finally regress to obtain 3D human posture information; S6: Perform supervised fine-tuning on the intermediate model: adjust the parameters of the regression head of the intermediate model, freeze other parameters of the intermediate model, and obtain the final 3D human pose estimation model.

2. A 3D human body posture estimation method based on angle map and image feature fusion as claimed in claim 1, characterized in that: The step S1 specifically includes: S1.1: Transform the two-dimensional images in the Human3.6M dataset The coordinates of 16 2D key points are obtained after normalization through the cascade pyramid network. ,in ; S1.2: Using the Laplacian of Gaussian Edge sharpening is performed , whose resolution is ; The Laplace operator of Gaussian is: （1） in, represents the standard deviation, , Represent the coordinates of the pixel points, Represents the natural logarithm.

3. The 3D human body posture estimation method based on angle map and image feature fusion as claimed in claim 1, characterized in that: The step S2 specifically includes: S2.1: The original skeleton graph with human joints as nodes Converted to a line graph with human bones as nodes ; in Represented as the original skeleton graph and line graph The node set and edge set of and , Line graph The nodes are the edges of the original skeleton graph, expressed as , and the edge is represented by ;Two edges in the original skeleton graph are connected if and only if they have a common node; S2.2: Calculate the original skeleton graph The angle between two edges with a common node in the graph is plotted as a line The weight above uses the coordinates of the two end points of the original skeleton graph as the line graph Node features ; in , the line graph The adjacency matrix of ,in , It can be obtained by the following formula: （2） in is a binary matrix, representing the node set in the original skeleton graph Is the node of the edge set The endpoints of the middle edge, is represented as the identity matrix, and F(·) is represented as the function for calculating the joint angles in the original skeleton graph.

4. The 3D human body posture estimation method based on angle map and image feature fusion as claimed in claim 1, characterized in that: The step S3 specifically includes: S3.1: The original skeleton graph Converted into a second-order graph that directly represents the angle relationship between joint points ; in , represents the edges between nodes with a path length of 2 in the original skeleton graph; S3.2: Compute the second-order graph The adjacency matrix of Generate node features ; The original skeleton graph The adjacency matrix of ,in , , a value of 1 indicates that the two nodes are connected, and a value of 0 indicates that they are not connected; a second-order graph representing the angle The adjacency matrix of It is expressed as: （3） Second-order graph Node features ,when and When is a second-order neighbor, and The angle value between or and When it is not a second-order neighbor, .

5. The 3D human body posture estimation method based on angle map and image feature fusion as claimed in claim 1, characterized in that: Step S4 specifically includes: S4.1: Use ordinary graph neural network layers to classify line graphs separately and the second-order graph Perform feature extraction and generate intermediate features and ,in is the feature dimension, and the original image Apply ordinary graph neural network to generate intermediate features ; The general graph neural network layer is represented as: （4） The above formula can be further described as: （5） （6） in, represents the learnable weight matrix, Line graph The adjacency matrix of represents the activation function, represents the modulation matrix, represents the Hadamard product, Represented by the second-order graph Or original image The adjacency matrix obtained after symmetric normalization of the adjacency matrix; S4.2: The intermediate features of the line graph are transformed into dimensions through the unpooling graph neural network. ; The depooling graph neural network layer includes two separate graph convolutional networks, which are used to obtain the embedding matrix and the alignment matrix respectively: （7） （8） in, represents the embedding matrix, represents the alignment matrix, and finally, the line graph The second-order graph is obtained by the following formula Middle layer features with consistent dimensions: (9) in, Represents the line graph after the anti-pooling operation The intermediate characteristics of S4.3: Fusion of multi-scale angle map features through cross-contrast learning; specifically, feature learning is constrained by three different forms of loss functions; Cross-compare original image features With line graph features , which is used to enhance the expression of local angle information and edge geometric characteristics. The calculation formula is: （10） in, Expressed as cosine similarity, Expressed as a temperature parameter, , Represented as vectors of nodes in the original graph and line graph respectively; Cross-compare original image features and second-order graph features , which is used to enhance the capture of high-order angle relationships and global structural information. The calculation formula is: （11） in, Represented as a node Eigenvectors in second-order graphs; Cross-comparison of original graph features in attribute module The original graph features in the structure module , used to ensure the consistency of multi-view features, the calculation formula is: （12） Through the joint constraints of the above contrast loss, the learnable weight parameters are used , fuse the features of the three views to generate intermediate fusion features , the fusion formula is: （13） in, Represented as a concatenation operation along the feature dimension.

6. The 3D human body posture estimation method based on angle map and image feature fusion as claimed in claim 1, characterized in that: Step S5 specifically includes: S5.1: Train the angle graph feature comparison and fusion module to obtain the intermediate model and optimize the loss function by gradient descent , the overall contrast loss of the angle map feature contrast fusion module is: （14） in, and Represented as a hyperparameter, used to balance the loss terms; S5.2: Input the processed image into a convolutional neural network model to obtain image features, and fuse the image features with the angle map features; the convolutional neural network model is HRNet, and the image features of different scales in four stages are fused with the angle map features through a conversion module, wherein the conversion module uses a fully connected layer to reduce the two-dimensional image features to one dimension.

7. The 3D human body posture estimation method based on angle map and image feature fusion as claimed in claim 1, characterized in that: The supervised fine-tuning process of the intermediate model in step S6 freezes other parameters of the intermediate model, and fine-tunes the output head of the prediction module using a weighted loss function. The specific formula is expressed as: （15） in, Represents the weighted sum of the mean square error and the mean absolute error, and the weighting coefficient .

8. A system for implementing a 3D human body posture estimation method based on angle map and image feature fusion as claimed in claim 1, characterized in that: It includes data preprocessing module, graph network construction module, angle graph feature comparison and fusion module, image feature fusion module and reasoning module; The data preprocessing module obtains the coordinates of 2D key points of the human body from the original image and performs normalization processing, and uses the Gaussian Laplace operator to enhance the edge features of the original image; The graph network construction module uses known 2D posture information to construct a network in two different ways, so that the new graph network structure has the ability to represent joint angle information; The angle feature comparison and fusion module cross-compares the multi-scale angle image features so that the angle image features of different scales have consistent dimensions and perform feature fusion; The image feature fusion module uses a fully connected layer network to fuse two different features, image and graph, at different stages of the model; The reasoning module finally outputs the reasoning result of the model, that is, the 3D posture information of the human body; The data preprocessing module, graph network construction module, angle graph feature comparison and fusion module, image feature fusion module and reasoning module are connected in sequence.

9. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the 3D human body posture estimation method based on the fusion of angle graph and image features described in any one of claims 1 to 7 is implemented.

Images