3D human posture estimation method and device based on graph convolution high-resolution network

Abstract

The present invention discloses a method and device for estimating a three-dimensional human posture based on a graph convolution high-resolution network, the method comprising: obtaining two-dimensional human key points to obtain a first intermediate feature; performing a graph convolution operation and a copy operation on the first intermediate feature to obtain a second intermediate feature and a third intermediate feature; performing a feature fusion operation according to the second intermediate feature and the third intermediate feature to obtain a body key point and a fourth intermediate feature; performing a residual connection according to the fourth intermediate feature and the first intermediate feature to obtain a right hand key point, a left hand key point and a facial key point; performing a splicing operation on the body key point, the right hand key point, the left hand key point and the facial key point to obtain a target feature; wherein the target feature is a three-dimensional human posture estimation coordinate. The present invention can improve the accuracy of three-dimensional human posture estimation and can be widely used in the field of posture estimation technology.

Description

Technical Field

[0001] The present invention relates to the technical field of posture estimation, and in particular to a three-dimensional human posture estimation method and device based on a graph convolution high-resolution network. Background Art

[0002] Existing full-body pose estimation technologies mainly rely on limited key point information, which limits the accurate capture of complex human poses. Especially in the estimation of fine movements of the face and hands, the traditional two-dimensional key point detection method cannot meet the needs of pose estimation in three-dimensional space, and cannot meet the needs of more refined full-body pose estimation, which greatly reduces the accuracy of full-body pose estimation. In the existing three-dimensional full-body pose estimation technology solutions, the multi-stage method based on separation modules has the problems of information loss and limited interaction between modules, resulting in a decrease in the accuracy of full-body pose estimation. Summary of the invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide a three-dimensional human posture estimation method and device based on a graph convolutional high-resolution network to improve the accuracy of three-dimensional human posture estimation.

[0004] To achieve the above object, an embodiment of the present invention provides a method for estimating a 3D human posture based on a graph convolutional high-resolution network, the method comprising:

[0005] Obtain two-dimensional human body key points and obtain the first intermediate features;

[0006] Performing a graph convolution operation and a copy operation on the first intermediate feature to obtain a second intermediate feature and a third intermediate feature;

[0007] Perform a feature fusion operation according to the second intermediate feature and the third intermediate feature to obtain body key points and a fourth intermediate feature;

[0008] Perform residual connection based on the fourth intermediate feature and the first intermediate feature to obtain right hand key points, left hand key points and facial key points;

[0009] Performing a splicing operation on the body key points, the right hand key points, the left hand key points and the facial key points to obtain target features;

[0010] Wherein, the target feature is the three-dimensional human body posture estimation coordinate.

[0011] In some embodiments, a 3D human body posture estimation method based on a graph convolutional high-resolution network further includes:

[0012] A feature fusion operation and a skeleton pooling operation are performed according to the second intermediate feature and the third intermediate feature to obtain a fifth intermediate feature.

[0013] In some embodiments, performing a graph convolution operation and a copy operation on the first intermediate feature to obtain a second intermediate feature and a third intermediate feature includes:

[0014] Inputting the first intermediate feature into the graph convolution block to perform a graph convolution operation to obtain a first output feature;

[0015] Inputting the first output feature into a basic block to obtain a second output feature;

[0016] Performing a copy operation on the second output feature to obtain a second intermediate feature and a third intermediate feature;

[0017] Wherein, the basic block includes several graph convolution blocks.

[0018] In some embodiments, performing a feature fusion operation according to the second intermediate feature and the third intermediate feature to obtain a body key point and a fourth intermediate feature includes:

[0019] Performing a skeleton pooling operation on the second intermediate feature to obtain a sixth intermediate feature;

[0020] Inputting the sixth intermediate feature into the graph convolution block and the basic block to obtain a third output feature;

[0021] Inputting the third intermediate feature into the basic block to obtain a fourth output feature;

[0022] Perform a feature fusion operation according to the third output feature and the fourth output feature to obtain a first fused feature and a second fused feature;

[0023] Several consecutive feature fusion operations are performed based on the first fusion feature and the second fusion feature to obtain body key points and a fourth intermediate feature.

[0024] In some embodiments, performing residual connection according to the fourth intermediate feature and the first intermediate feature to obtain right hand key points, left hand key points and facial key points includes:

[0025] Inputting the first intermediate feature into the graph convolution block to perform a graph convolution operation to obtain a first output feature;

[0026] Performing a residual connection on the first output feature and the fourth intermediate feature to obtain a residual feature;

[0027] The residual features are input into three basic blocks respectively to obtain right hand key points, left hand key points and facial key points.

[0028] In some embodiments, performing a feature fusion operation and a skeleton pooling operation according to the second intermediate feature and the third intermediate feature to obtain a fifth intermediate feature includes:

[0029] Performing a skeleton pooling operation on the second intermediate feature to obtain a sixth intermediate feature;

[0030] Inputting the sixth intermediate feature into the graph convolution block and the basic block to obtain a third output feature;

[0031] Inputting the third intermediate feature into the basic block to obtain a fourth output feature;

[0032] Perform a feature fusion operation according to the third output feature and the fourth output feature to obtain a first fusion feature;

[0033] The first fusion feature is subjected to a skeleton pooling operation to obtain a fifth intermediate feature.

[0034] To achieve the above object, another aspect of an embodiment of the present invention provides a 3D human posture estimation device based on a graph convolution high-resolution network, the device comprising:

[0035] The first module is used to obtain two-dimensional human body key points and obtain the first intermediate features;

[0036] A second module is used to perform a graph convolution operation and a copy operation on the first intermediate feature to obtain a second intermediate feature and a third intermediate feature;

[0037] A third module is used to perform a feature fusion operation according to the second intermediate feature and the third intermediate feature to obtain a body key point and a fourth intermediate feature;

[0038] A fourth module is used to perform residual connection according to the fourth intermediate feature and the first intermediate feature to obtain right hand key points, left hand key points and facial key points;

[0039] A fifth module is used to perform a splicing operation on the body key points, the right hand key points, the left hand key points and the facial key points to obtain target features;

[0040] Wherein, the target feature is the three-dimensional human body posture estimation coordinate.

[0041] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the above-mentioned three-dimensional human posture estimation method based on a graph convolution high-resolution network.

[0042] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned three-dimensional human posture estimation method based on a graph convolution high-resolution network.

[0043] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the aforementioned three-dimensional human posture estimation method based on a graph convolution high-resolution network.

[0044] Embodiments of the present invention include at least the following beneficial effects: The present application provides a method and device for estimating three-dimensional human body posture based on a graph convolution high-resolution network, the scheme acquires two-dimensional human body key points to obtain a first intermediate feature; performs a graph convolution operation and a copy operation on the first intermediate feature to obtain a second intermediate feature and a third intermediate feature; performs a feature fusion operation based on the second intermediate feature and the third intermediate feature to obtain body key points and a fourth intermediate feature; performs a residual connection based on the fourth intermediate feature and the first intermediate feature to obtain right hand key points, left hand key points and facial key points; performs a splicing operation on the body key points, the right hand key points, the left hand key points and the facial key points to obtain a target feature; wherein the target feature is a three-dimensional human body posture estimation coordinate, the scheme uses a residual connection to fuse global and local information for the body key points, the right hand key points, the left hand key points and the facial key points, performs a fine regression prediction on the three-dimensional coordinates of these fine parts, and improves the accuracy of three-dimensional human body posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 It is a flow chart of a three-dimensional human posture estimation method based on a graph convolution high-resolution network provided in an embodiment of the present application;

[0047] Figure 2 It is a schematic diagram of skeleton pooling and skeleton de-pooling provided in an embodiment of the present application;

[0048] Figure 3 is a schematic diagram of 133 whole-body bone point numbers provided in an embodiment of the present application;

[0049] Figure 4 It is a schematic diagram of the structure of a graph convolutional high-resolution network provided in an embodiment of the present application;

[0050] Figure 5 It is a structural schematic diagram of a three-dimensional human posture estimation device based on a graph convolution high-resolution network provided in an embodiment of the present application;

[0051] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0053] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0054] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] Through the analysis of existing 3D full-body pose estimation technology solutions, we can find the following main shortcomings: the multi-stage method based on separated modules has the problems of information loss and limited interaction between modules, resulting in reduced estimation accuracy; the multi-stage structure makes the overall model of 3D full-body pose estimation complex, with a large number of parameters and slow inference speed; the one-stage method based on graph convolutional network is limited by the limited receptive field of the convolution kernel. When the human body posture structure is complex, the information transmission between different parts will be hindered. Existing 3D full-body pose estimation methods are difficult to simultaneously and precisely estimate the torso, face and hands with large scale differences.

[0057] In view of this, if Figure 1 As shown, an embodiment of the present invention provides a 3D human posture estimation method based on a graph convolution high-resolution network. Figure 1 The method may include but is not limited to steps S100 to S500.

[0058] Step S100, acquiring two-dimensional human body key points to obtain a first intermediate feature;

[0059] Step S200, performing a graph convolution operation and a copy operation on the first intermediate feature to obtain a second intermediate feature and a third intermediate feature;

[0060] Step S300, performing a feature fusion operation according to the second intermediate feature and the third intermediate feature to obtain body key points and a fourth intermediate feature;

[0061] Step S400, performing residual connection according to the fourth intermediate feature and the first intermediate feature to obtain right hand key points, left hand key points and facial key points;

[0062] Step S500, performing a splicing operation on the body key points, the right hand key points, the left hand key points and the facial key points to obtain target features.

[0063] In some embodiments, the three-dimensional human posture estimation method based on graph convolution high-resolution network may also include but is not limited to step S600:

[0064] Step S600, performing a feature fusion operation and a skeleton pooling operation according to the second intermediate feature and the third intermediate feature to obtain a fifth intermediate feature.

[0065] In step S100 of some embodiments, an RGB image containing the entire human body can be obtained through a monocular camera, and an existing two-dimensional human body posture estimation algorithm can be used to obtain two-dimensional human body key points to obtain a first intermediate feature with a dimension of (133,2).

[0066] In some embodiments, step S200 may include but is not limited to steps S210 to S230:

[0067] Step S210, inputting the first intermediate feature into a graph convolution block to perform a graph convolution operation to obtain a first output feature;

[0068] Step S220, inputting the first output feature into a basic block to obtain a second output feature;

[0069] Step S230, copying the second output feature to obtain a second intermediate feature and a third intermediate feature; wherein the basic block includes a plurality of graph convolution blocks.

[0070] Specifically, graph convolution is an extended form of traditional convolutional neural network when processing non-Euclidean structure data (such as graph structure data). Its basic idea is to learn how to combine the features of node neighborhoods on the graph to obtain a higher-level node representation. The relationship between human skeleton points can be represented by graph structure data. The embodiment of the present invention applies the graph convolution operation to three-dimensional human posture estimation to obtain the key points of various parts of the human body. In addition, graph convolution is a neural network layer. The transmission method between layers is to first convert the input feature matrix H (l) With the renormalized adjacency matrix and convolution kernel W (l) Multiply them to perform linear transformation, and then use the nonlinear activation function σ(·) to perform nonlinear activation on the result of the linear transformation to obtain the feature matrix H of the output activation of the graph convolution at the l+1th layer (l+1) . Then we have the following expression:

[0071]

[0072] In the formula,

[0073] in, represents the self-loop matrix; A represents the adjacency matrix; I represents the identity matrix; H represents the features of each layer; σ(·) represents the nonlinear activation function; W represents the weight matrix of each layer; l+1, l represent the number of layers; Represents the degree matrix of the self-loop matrix, then we have the formula Represents the i-th element on the diagonal of the degree matrix; Represents the element in the i-th row and j-th column of the self-loop matrix. Through the above graph convolution, the features of each layer continuously capture the information of adjacent nodes for updating. Applying the graph convolution operation to 3D human pose estimation can obtain the 3D human pose estimation coordinates.

[0074] like Figure 4As shown, in the high-resolution module, in the first-level branch 701, the first intermediate feature with a dimension of (133, 2) is input into the graph convolution block for graph convolution operation, and the first output feature with a dimension of (133, 32) can be obtained. The first output feature is input into the basic block for several graph convolution operations to obtain the second output feature, wherein the basic block includes several graph convolution blocks. The second output feature is copied to obtain the second intermediate feature and the third intermediate feature, and the third intermediate feature is left in the first-level branch 701 for subsequent processing, while the second intermediate feature is used to obtain the feature of the second-level branch 702, which will be described in detail later.

[0075] In some embodiments, step S300 may include but is not limited to steps S310 to S350:

[0076] Step S310, performing a skeleton pooling operation on the second intermediate feature to obtain a sixth intermediate feature;

[0077] Step S320, inputting the sixth intermediate feature into the graph convolution block and the basic block to obtain a third output feature;

[0078] Step S330, inputting the third intermediate feature into the basic block to obtain a fourth output feature;

[0079] Step S340, performing a feature fusion operation according to the third output feature and the fourth output feature to obtain a first fused feature and a second fused feature;

[0080] Step S350, performing several consecutive feature fusion operations based on the first fusion feature and the second fusion feature to obtain body key points and a fourth intermediate feature.

[0081] Specifically, refer to Figure 2 Skeletal Unpooling is an upsampling operation that restores the low-scale skeleton structure to its original size by copying the node features in the low-scale skeleton structure map as the corresponding two nodes in the high-scale skeleton structure map. The skeleton unpooling operation enables the model to fuse higher-scale skeleton information and pass it to subsequent processing layers. Exemplarily, the skeleton unpooling operation is performed on the data set H3WB, where the data set H3WB has 133 skeleton points, and the specific node numbers are as follows: Figure 3 There are 35 nodes in the third-level branch. Copying these 35 nodes in the following pattern will yield 68 nodes in the second-level branch:

[0082] [1,1,2,2,3,3,4,4,5,5,6,6,7,8,9,10,11,10,9,8,7,12,13,14,13,12,14,15,16,15,16,17,18,19,17,18,19,20,11,20,21,21,22,22,23,23,24,25,25,26,26,27,27,28,28,29,29,30,31,31,32,32,33,33,34,34,35,35]

[0083] The value of the i-th element in this list is j, which means that the feature of the j-th node of the third-level branch is assigned to the j-th node of the second-level branch.

[0084] Copy the second-level nodes in the following mode to obtain 133 full-body bone points of the first-level branch:

[0085] [3,1,1,2,2,4,4,6,5,6,5,8,7,8,7,10,9,12,12,10,11,11,9,13,13,14,14,15,15,16,16,17,18,18,19,19,20,20,21,21,22,22,23,23,24,24,25,25,26,26,27,27,28,28,29,29,30,31,31,32,33,33,32,34,34,35,36,36,35, 37,37,38,38,39,30,39,40,40,41,41,17,42,42,43,43,44,44,45,45,46,46,47,48,48,49,49,50,50,51,51,52,52,53,53,54,54,55,55,56,56,57,57,58,59,59,60,60,61,61,62,62,63,63,64,64,65,65,66,66,67,67,68,68]

[0086] The value of the i-th element in this list is j, which means that the feature of the j-th node of the second-level branch is assigned to the j-th node of the first-level branch.

[0087] like Figure 2As shown in the figure, skeleton pooling is a downsampling operation. According to the characteristics of the human body structure, human nodes are grouped in pairs, and the features of these corresponding nodes are fused into a single node in the low-scale skeleton structure diagram through average pooling operation. That is, skeleton pooling is to sum and average the node features of the previous level branch according to the above correspondence to obtain the node features of the next level. The skeleton pooling operation not only utilizes the topological structure of the human skeleton, but also greatly reduces the computational complexity of the pooling layer due to its simplicity, and improves the speed of training and reasoning.

[0088] like Figure 4 As shown, in the high-resolution module (High-Resolution Module), the second intermediate feature is subjected to the skeleton pooling operation, and the sixth intermediate feature with a dimension of (68, 64) is obtained. The sixth intermediate feature is input into the graph convolution block in the second-level branch 702 and the basic block is subjected to the graph convolution operation, and the third output feature is obtained. In the first-level branch 701, the third intermediate feature is input into the basic block, and the fourth output feature is obtained. Then, according to the third output feature and the fourth output feature, the first-level branch 701 and the second-level branch 702 are subjected to feature fusion, that is, the fourth output feature is subjected to the skeleton pooling operation, and the fourth output feature after the skeleton pooling process is input into the second-level branch 702, and the feature fusion operation is performed with the third output feature, and the first fusion feature is obtained. In addition, the third output feature is subjected to the skeleton anti-pooling operation, and the third output feature after the skeleton anti-pooling process is input into the first-level branch 701, and the feature fusion operation is performed with the fourth output feature, and the second fusion feature is obtained.

[0089] According to the first fusion feature and the second fusion feature, several continuous feature fusion operations are performed on the first-level branch 701 and the second-level branch 702. The feature fusion process is similar to the process of performing feature fusion on the first-level branch 701 and the second-level branch 702 according to the third output feature and the fourth output feature, including: in the second-level branch 702, the first fusion feature is input into the basic block for processing to obtain the fifth output feature; in the first-level branch 701, the second fusion feature is input into the basic block to obtain the sixth output feature; the skeleton anti-pooling operation is performed on the fifth output feature, the fifth output feature after the skeleton anti-pooling processing is input into the first branch 701, and the feature is fused with the sixth output feature to obtain the third fusion feature; the skeleton pooling operation is performed on the sixth output feature, the sixth output feature after the skeleton pooling processing is input into the second-level branch 702, and the feature is fused with the fifth output feature to obtain the fourth fusion feature. Similarly, according to the third fusion feature and the fourth fusion feature, the first level branch 701 and the second level branch 702 are subjected to feature fusion operation, and finally the first level branch 701 outputs the fourth intermediate feature and the body key points with the dimension of (23, 3). By performing several consecutive feature fusion operations on the first level branch 701 and the second level branch, each level branch can capture the features of other branches, thereby overcoming the problem of limited receptive field of graph convolution.

[0090] In some embodiments, step S400 may include but is not limited to steps S410 to S430:

[0091] Step S410, inputting the first intermediate feature into a graph convolution block to perform a graph convolution operation to obtain a first output feature;

[0092] Step S420, performing residual connection on the first output feature and the fourth intermediate feature to obtain a residual feature;

[0093] Step S430, inputting the residual features into three basic blocks respectively to obtain right hand key points, left hand key points and facial key points.

[0094] Specifically, Figure 4As shown, the fourth intermediate feature including the lower body feature of the human body is input into the multi-branch regression module, and three parallel branches are set in the multi-branch regression module, which are respectively used to refine the three-dimensional coordinates of the right hand, the left hand and the face. In the first-level branch 701, the first intermediate feature with a dimension of (133, 2) is input into the graph convolution block for graph convolution operation, and the first output feature with a dimension of (133, 32) can be obtained. The first output feature is residually connected with the fourth intermediate feature to obtain the residual feature, and the residual feature is sent to the three basic blocks set in the multi-branch regression module to obtain the right hand key point with a dimension of (21, 3), the left hand key point with a dimension of (21, 3) and the face key point with a dimension of (68, 3). Among them, the feature dimensions processed by the three basic blocks are 21, 21 and 68 respectively, and the network structures of the three basic blocks are almost the same. Through the three basic blocks, the fine-grained features of the hand and face are more refined. The coordinate regression is performed on the fine-grained features of the hand and face to obtain the final three-dimensional coordinates of the hand and face.

[0095] The embodiment of the present invention constructs a graph convolution high-resolution network (Graph High-Resolution Network), which includes a high-resolution module (High-Resolution Module) and a multi-branch regression module (Multi-branch Regression Module). Through the hierarchical branch structure of the high-resolution module and the design of the multi-branch regression module, the three-dimensional coordinates of the human body parts are finely predicted according to the fusion features. The graph convolution high-resolution network can capture global structural information and local detail information at the same time, thereby completing high-precision three-dimensional human posture estimation. In particular, for the hands and face parts with large scale changes, the regression estimation effect using special branches is better.

[0096] In step S500 of some embodiments, a splicing operation is performed on the body key points, the right hand key points, the left hand key points and the facial key points to obtain a target feature with a dimension of (133, 3). The target feature is a three-dimensional human body posture estimation coordinate, which includes 133 3D whole body joint point coordinates.

[0097] In some embodiments, step S600 may include but is not limited to steps S610 to S650:

[0098] Step S610, performing a skeleton pooling operation on the second intermediate feature to obtain a sixth intermediate feature;

[0099] Step S620, inputting the sixth intermediate feature into the graph convolution block and the basic block to obtain a third output feature;

[0100] Step S630, inputting the third intermediate feature into the basic block to obtain a fourth output feature;

[0101] Step S640, performing a feature fusion operation according to the third output feature and the fourth output feature to obtain a first fused feature;

[0102] Step S650: performing a skeleton pooling operation on the first fusion feature to obtain a fifth intermediate feature.

[0103] Specifically, Figure 4 As shown, the second intermediate feature is subjected to a skeleton pooling operation to obtain a sixth intermediate feature of dimension (68, 64), and the sixth intermediate feature is input into the graph convolution block and the basic block in the second-level branch 702 to perform a graph convolution operation to obtain a third output feature. In the first-level branch 701, the third intermediate feature is input into the basic block to obtain a fourth output feature. Then, according to the third output feature and the fourth output feature, feature fusion is performed on the first-level branch 701 and the second-level branch 702, that is, the fourth output feature is subjected to a skeleton pooling operation, and the fourth output feature after the skeleton pooling process is input into the second-level branch 702, and a feature fusion operation is performed with the third output feature to obtain a first fused feature. The first fused feature is subjected to a skeleton pooling operation to obtain a fifth intermediate feature of dimension (35, 128), and the fifth intermediate feature is input into the graph convolution block of the third branch 703 to perform a graph convolution operation to obtain a fifth fused feature. According to the first fusion feature and the fifth fusion feature, several consecutive feature fusion operations are performed on the second branch 702 and the third branch 703. The feature fusion process is similar to the above-mentioned process of performing several consecutive feature fusion operations on the first-level branch 701 and the second-level branch according to the first fusion feature and the second fusion feature.

[0104] In the process of several consecutive feature fusions between the first-level branch 701 and the second-level branch 702, and the process of several consecutive feature fusions between the second branch 702 and the third branch 703, each feature fusion process fuses the features of the first-level and second-level branches in the first-level branch, fuses the features of the second-level and third-level branches in the third-level branch, and fuses the features of the first-level, second-level and third-level branches in the second-level branch. After multiple consecutive feature fusions, each level of branch can capture the features of other branches, thereby overcoming the problem of limited receptive field of graph convolution.

[0105] like Figure 5 As shown, the embodiment of the present invention further provides a 3D human body posture estimation device 700 based on a graph convolution high-resolution network, which can implement the above-mentioned 3D human body posture estimation method based on a graph convolution high-resolution network, and the device includes:

[0106] The first module 701 is used to obtain two-dimensional human body key points and obtain a first intermediate feature;

[0107] The second module 702 is used to perform a graph convolution operation and a copy operation on the first intermediate feature to obtain a second intermediate feature and a third intermediate feature;

[0108] The third module 703 is used to perform a feature fusion operation according to the second intermediate feature and the third intermediate feature to obtain a body key point and a fourth intermediate feature;

[0109] A fourth module 704 is used to perform residual connection according to the fourth intermediate feature and the first intermediate feature to obtain right hand key points, left hand key points and facial key points;

[0110] The fifth module 705 is used to perform a splicing operation on the body key points, the right hand key points, the left hand key points and the facial key points to obtain target features;

[0111] Wherein, the target feature is the three-dimensional human body posture estimation coordinate.

[0112] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0113] The embodiment of the present invention further provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned three-dimensional human posture estimation method based on a graph convolution high-resolution network is implemented. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0114] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0115] refer to Figure 6 , Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0116] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0117] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes a three-dimensional human posture estimation method based on a graph convolution high-resolution network in the embodiments of this application;

[0118] Input / output interface 903, used to implement information input and output;

[0119] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0120] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0121] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0122] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned three-dimensional human posture estimation method based on a graph convolution high-resolution network.

[0123] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0124] The embodiment of the present invention also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the aforementioned three-dimensional human posture estimation method based on a graph convolution high-resolution network.

[0125] In summary, a 3D human body posture estimation method and device based on a graph convolution high-resolution network according to an embodiment of the present invention has the following advantages:

[0126] 1. The embodiment of the present invention constructs an end-to-end single-stage framework, which avoids the information loss and module interaction limitation problems of the multi-stage method, and is conducive to improving the estimation accuracy.

[0127] 2. The embodiment of the present invention constructs a graph convolution high-resolution network, expands the receptive field of the convolution kernel, promotes the efficient fusion and information transmission of features of various parts of the body, and at the same time, adopts a multi-branch structure to process features of different scales respectively, so that the graph convolution high-resolution network can simultaneously perform fine estimation of different parts such as the torso, face and hands, thereby improving the accuracy and efficiency of 3D whole-body posture estimation.

[0128] 3. The embodiment of the present invention integrates an enhanced graph convolution kernel through a high-resolution module, and adopts a hierarchical parallel structure to effectively capture multi-scale feature information, thereby solving the problem of limited receptive field of traditional graph convolution.

[0129] 4. The embodiment of the present invention uses a multi-branch regression module, including special branches for detailed parts such as hands and faces, and uses residual connections to fuse global and local information to perform fine regression prediction on the 3D coordinates of fine parts.

[0130] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0131] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0132] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0133] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0134] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0135] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0136] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0137] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0138] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A three-dimensional human posture estimation method based on graph convolutional high-resolution network, characterized in that: include: Obtain two-dimensional human body key points and obtain the first intermediate features; Performing a graph convolution operation and a copy operation on the first intermediate feature to obtain a second intermediate feature and a third intermediate feature; Perform a feature fusion operation according to the second intermediate feature and the third intermediate feature to obtain body key points and a fourth intermediate feature; Perform residual connection based on the fourth intermediate feature and the first intermediate feature to obtain right hand key points, left hand key points and facial key points; Performing a splicing operation on the body key points, the right hand key points, the left hand key points and the facial key points to obtain target features; Wherein, the target feature is a three-dimensional human body posture estimation coordinate; The performing a graph convolution operation and a copy operation on the first intermediate feature to obtain a second intermediate feature and a third intermediate feature includes: Inputting the first intermediate feature into the graph convolution block to perform a graph convolution operation to obtain a first output feature; Inputting the first output feature into a basic block to obtain a second output feature; Performing a copy operation on the second output feature to obtain a second intermediate feature and a third intermediate feature; Wherein, the basic block includes a plurality of graph convolution blocks; The step of performing a feature fusion operation according to the second intermediate feature and the third intermediate feature to obtain a body key point and a fourth intermediate feature includes: Performing a skeleton pooling operation on the second intermediate feature to obtain a sixth intermediate feature; Inputting the sixth intermediate feature into the graph convolution block and the basic block to obtain a third output feature; Inputting the third intermediate feature into the basic block to obtain a fourth output feature; Perform a feature fusion operation according to the third output feature and the fourth output feature to obtain a first fused feature and a second fused feature; Performing several consecutive feature fusion operations based on the first fusion feature and the second fusion feature to obtain body key points and a fourth intermediate feature; The performing residual connection according to the fourth intermediate feature and the first intermediate feature to obtain right hand key points, left hand key points and facial key points includes: Inputting the first intermediate feature into the graph convolution block to perform a graph convolution operation to obtain a first output feature; Performing a residual connection on the first output feature and the fourth intermediate feature to obtain a residual feature; The residual features are input into three basic blocks respectively to obtain right hand key points, left hand key points and facial key points.

2. The method for 3D human posture estimation based on graph convolutional high-resolution network according to claim 1, characterized in that: Also includes: A feature fusion operation and a skeleton pooling operation are performed according to the second intermediate feature and the third intermediate feature to obtain a fifth intermediate feature.

3. The method for 3D human posture estimation based on graph convolutional high-resolution network according to claim 2, characterized in that: The performing a feature fusion operation and a skeleton pooling operation according to the second intermediate feature and the third intermediate feature to obtain a fifth intermediate feature includes: Performing a skeleton pooling operation on the second intermediate feature to obtain a sixth intermediate feature; Inputting the sixth intermediate feature into the graph convolution block and the basic block to obtain a third output feature; Inputting the third intermediate feature into the basic block to obtain a fourth output feature; Perform a feature fusion operation according to the third output feature and the fourth output feature to obtain a first fusion feature; The first fusion feature is subjected to a skeleton pooling operation to obtain a fifth intermediate feature.

4. A 3D human posture estimation device based on graph convolution high-resolution network, characterized in that: include: The first module is used to obtain two-dimensional human body key points and obtain the first intermediate features; A second module is used to perform a graph convolution operation and a copy operation on the first intermediate feature to obtain a second intermediate feature and a third intermediate feature; A third module is used to perform a feature fusion operation according to the second intermediate feature and the third intermediate feature to obtain a body key point and a fourth intermediate feature; A fourth module is used to perform residual connection according to the fourth intermediate feature and the first intermediate feature to obtain right hand key points, left hand key points and facial key points; A fifth module is used to perform a splicing operation on the body key points, the right hand key points, the left hand key points and the facial key points to obtain target features; Wherein, the target feature is a three-dimensional human body posture estimation coordinate; The second module is specifically used for: Inputting the first intermediate feature into the graph convolution block to perform a graph convolution operation to obtain a first output feature; Inputting the first output feature into a basic block to obtain a second output feature; Performing a copy operation on the second output feature to obtain a second intermediate feature and a third intermediate feature; Wherein, the basic block includes a plurality of graph convolution blocks; The third module is specifically used for: Performing a skeleton pooling operation on the second intermediate feature to obtain a sixth intermediate feature; Inputting the sixth intermediate feature into the graph convolution block and the basic block to obtain a third output feature; Inputting the third intermediate feature into the basic block to obtain a fourth output feature; Perform a feature fusion operation according to the third output feature and the fourth output feature to obtain a first fused feature and a second fused feature; Performing several consecutive feature fusion operations based on the first fusion feature and the second fusion feature to obtain body key points and a fourth intermediate feature; The fourth module is specifically used for: Inputting the first intermediate feature into the graph convolution block to perform a graph convolution operation to obtain a first output feature; Performing a residual connection on the first output feature and the fourth intermediate feature to obtain a residual feature; The residual features are input into three basic blocks respectively to obtain right hand key points, left hand key points and facial key points.

5. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images