A monocular three-dimensional human pose estimation method and system fusing space-time features

Abstract

The application relates to a monocular three-dimensional human posture estimation method and system fusing space-time features, which fuses local and global space information and combines time information, and the method comprises the following steps: performing two-dimensional human posture estimation according to a three-dimensional human posture estimation data set, and lifting the obtained two-dimensional human posture to a high-dimensional space; extracting local and global features of key points through a Transformer, fusing the local and global features, sending the fused features into a space global Transformer encoder, and extracting information in time sequence; and sending the information in time sequence into a single-layer perception machine decoder to obtain final three-dimensional human posture key points. The application is used for processing three-dimensional human posture estimation of monocular video, combines time sequence information, greatly suppresses the jitter of output, and can be widely applied in the field of computer vision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, especially the field of human pose estimation, and in particular to a monocular three-dimensional human pose estimation method and system based on deep learning by fusing local and global features. BACKGROUND

[0002] Human pose estimation is one of the important tasks of computer vision. Since the 1960s, research on human pose estimation has begun to appear in the field of computer vision. In recent years, with the development of artificial intelligence, human pose estimation methods have become more accurate, real-time, and have a wider range of research. Human pose estimation can be divided into two-dimensional human pose estimation and three-dimensional human pose estimation according to the format of the data. Three-dimensional human pose estimation can be divided into monocular three-dimensional human pose estimation and multi-view three-dimensional human pose estimation according to the dimension of the perspective. Most existing human pose estimation methods are two-dimensional human pose estimation methods based on neural networks, such as HRNet, CPN, openpose, etc. Although two-dimensional human pose estimation has challenges such as self-occlusion, lighting, and incomplete pose, existing two-dimensional human pose estimation methods can estimate two-dimensional human poses in most scenes. Compared with two-dimensional human pose estimation methods, three-dimensional human pose estimation methods have developed relatively slowly. First, three-dimensional human pose estimation adds a depth dimension, greatly increasing the complexity of pose estimation. Second, three-dimensional human pose datasets are relatively less than two-dimensional human pose datasets, which also increases the difficulty of developing three-dimensional human pose estimation methods. Finally, the lack of real three-dimensional pose data also slows down the development of three-dimensional human pose estimation methods. Existing three-dimensional human pose estimation methods mainly have two ideas: the first is to directly estimate three-dimensional human poses from image data, and the second is to enhance two-dimensional human skeletons to three-dimensional space from existing two-dimensional human pose methods. Compared with the first method, the second method is simpler and more efficient. Compared with skeleton data, image data contains more redundant information, which has a great impact on deep learning methods. Therefore, most three-dimensional human pose estimation methods in recent years are based on the method of enhancing two-dimensional skeletons to three-dimensional skeletons.

[0003] Early three-dimensional human posture methods mostly promote two-dimensional human posture through multi-view triangulation and other methods. Because of the existence of multi-view information, the method of promoting from two-dimensional skeleton to three-dimensional skeleton can combine depth information. However, multi-view data acquisition is difficult, and the application scene is more complex, so the application type of single-view three-dimensional human posture estimation method is more widely used. However, single-view three-dimensional human posture estimation has the problem of semantic defects, so it is difficult to obtain three-dimensional human posture from a single picture or monocular video, and some methods propose to use a deep neural network to promote two-dimensional human posture to three-dimensional space.

[0004] However, the method of promoting two-dimensional human posture through a deep neural network still has the following problems: (1) the network structure is too simple, and it is difficult to promote complex postures; (2) the output three-dimensional posture has large jitter; (3) local information and global information cannot be combined; (4) temporal information cannot be combined well.

[0005] In summary, it is particularly important to study a method that can obtain three-dimensional human posture by combining local and global information, and in addition, temporal information needs to be combined to suppress the jitter of the output. Therefore, a new method is needed to further improve the accuracy of three-dimensional human posture estimation. SUMMARY

[0006] In view of the above problems, the purpose of the present application is to provide a monocular three-dimensional human posture estimation method and system fusing spatiotemporal features, which is used for processing three-dimensional human posture estimation of monocular video, and combines temporal information, greatly suppressing the jitter of the output.

[0007] To achieve the above purpose, the present application adopts the following technical scheme: a monocular three-dimensional human posture estimation method fusing spatiotemporal features, which fuses local and global spatial information and combines temporal information, comprising: estimating two-dimensional human posture according to a three-dimensional human posture estimation dataset, and promoting the obtained two-dimensional human posture to a high-dimensional space; extracting local and global features of key points through a Transformer, fusing the local and global features, and sending the fused features into a spatial global Transformer encoder to extract temporal information; and sending the temporal information into a single-layer perception machine decoder to obtain the final three-dimensional human posture key points.

[0008] Further, the two-dimensional human posture estimation according to the three-dimensional human posture estimation dataset comprises: preprocessing the three-dimensional human posture estimation dataset to obtain a picture sequence including two-dimensional human posture and corresponding three-dimensional human posture information; sending the two-dimensional human posture picture sequence into a two-dimensional human posture detector to obtain a joint skeleton of two-dimensional human posture.

[0009] Further, the obtained two-dimensional human posture is lifted to a high-dimensional space, comprising:

[0010] The obtained two-dimensional human posture sequence is evenly divided into m sequences, each of which has f frames, and each frame contains two-dimensional spatial coordinates of 17 key points, that is, the skeleton sequence is divided into matrix sequence, and then each x i is the two-dimensional human key point coordinates of each frame; wherein J is the number of key points, and 2 is the two-dimensional coordinates of the key points;

[0011] The two-dimensional spatial point coordinates are lifted to a c-dimensional feature space, and coordinate coding is performed to obtain a feature matrix

[0012] Further, the features of the key points are extracted by the Transformer, comprising:

[0013] The transformed two-dimensional human skeleton sequence is divided into an upper half limb sequence Jup and a lower half limb sequence Jdown, and Jup and Jdown are respectively sent to the local information encoder, and J is sent to the global spatial key point Transformer encoder; wherein J=Jup+Jdown.

[0014] Further, the Transformer encoder comprises a multi-head attention mechanism and a multi-layer perception machine;

[0015] In the calculation of the multi-head attention mechanism, the attention of each head is calculated first, and the features of each head are fused to obtain the final multi-head attention;

[0016] After the multi-head attention mechanism, the upper half spatio-temporal self-attention feature matrix and the lower half spatio-temporal self-attention feature matrix are obtained, and the features of the upper and lower two feature matrices are sent to the regularization layer to regularize the data;

[0017] Through the multi-layer perception mechanism, the states of the two parts of spatio-temporal self-attention are updated separately to obtain the spatio-temporal feature matrix F(Z up ), and regularized.

[0018] Further, the information on the time sequence is extracted, comprising:

[0019] The obtained each frame and are fused, and f frames in the adjacent range are taken to form a sequence;

[0020] The feature matrix is deformed to obtain the final global feature matrix Z∈Rf×(J×c) The feature matrix is input into a space-time encoder similar to the above-mentioned Transformer encoder to extract space-time features, and a final matrix containing time and global features is obtained

[0021] Further, the information on the time sequence is input into a single-layer perception machine decoder to obtain the final three-dimensional human pose key points, including:

[0022] One-dimensional convolution is used to reduce the f frame features to one frame to generate the key point feature matrix of the intermediate frame

[0023] A single-layer perception machine is used as a decoder to reduce the c-dimensional feature matrix in the feature space to three dimensions:

[0024] F(TZ)=G(b (1) +W (1) TZ)

[0025] Wherein, G is a sigmoid activation function, and after matrix transformation, the final three-dimensional human pose is obtained

[0026] The loss function L of the network is calculated:

[0027]

[0028] Wherein, p k is the real three-dimensional key point coordinates, is the estimated three-dimensional human pose coordinates, wherein J is the number of all human key points, and k is the kth key point.

[0029] A monocular three-dimensional human pose estimation system fusing space-time features, which fuses local and global spatial information and combines time information, includes: a two-dimensional human pose estimation module, which estimates two-dimensional human poses according to a three-dimensional human pose estimation dataset, and elevates the obtained two-dimensional human poses to a high-dimensional space; a feature fusion module, which extracts key point local and global features through a Transformer, fuses the local and global features, inputs the fused features into a space global Transformer encoder, and extracts information on the time sequence; and a pose key point acquisition module, which inputs the information on the time sequence into a single-layer perception machine decoder to obtain the final three-dimensional human pose key points.

[0030] A computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform any of the above methods.

[0031] A computing device comprising: one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured for execution by the one or more processors, the one or more programs including instructions for performing any of the above methods.

[0032] The present application has the following advantages due to the above technical solutions:

[0033] 1. The present application improves the detection performance of monocular three-dimensional human pose estimation method.

[0034] 2. The present application adds an encoder combining global and local, time and space, adopts a deep learning method of lifting monocular video two-dimensional human pose to three-dimensional human pose, further improves the detection accuracy by fusing global and local information, time and space information, and improves the anti-shake performance to a certain extent.

[0035] 3. The present application improves the detection accuracy on the basis of existing two-dimensional human pose detector under the premise of ensuring speed, and can be more widely applied in actual production and life, and can be applied in animation driving, behavior recognition, motion capture and other scenes. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is the whole process schematic diagram of the estimation method in the embodiment of the present application;

[0037] Figure 2 is the monocular three-dimensional human pose estimation method flowchart in the embodiment of the present application;

[0038] Figure 3 is the whole network structure schematic diagram in the embodiment of the present application;

[0039] Figure 4 is the detection result schematic diagram in the embodiment of the present application;

[0040] Figure 5 is the computing device structure schematic diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0041] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0042] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0043] The application provides a monocular three-dimensional human pose estimation method and system fusing space-time features, which comprises: a monocular three-dimensional human pose estimation method and system fusing local and global, time and space, which uses a two-dimensional human pose estimation network CPN, a local feature encoder for extracting local space features, an encoder for fusing local space features and global features, a space feature encoder and a multi-layer perception decoder.

[0044] First, the two-dimensional human pose is estimated, then the obtained two-dimensional pose is lifted to a high-dimensional space, then the local and global features of the key points are extracted through the Transformer, then the two kinds of features are fused, then the fused features are sent into the space global Transformer encoder to extract the information in time sequence, and finally the final three-dimensional human pose is obtained through a single-layer neural network linear layer. The application proposes to estimate the three-dimensional human pose by combining the features of time and space, local and global, which improves the accuracy of three-dimensional human pose estimation and reduces the jitter between different frames.

[0045] In one embodiment of the application, a monocular three-dimensional human pose estimation method fusing space-time features is provided, which is specifically a method applied to lifting two-dimensional human key points to a three-dimensional space. In this embodiment, the method is exemplified by being applied to a terminal. It should be understood that the method can also be applied to a server, and can also be applied to a system comprising a terminal and a server, and can be realized through the interaction of the terminal and the server. In this embodiment, the local and global space information is fused, and the time information is combined, as shown in Figure 1 The method comprises the following steps:

[0046] 1) estimating the two-dimensional human pose according to the three-dimensional human pose estimation dataset, and lifting the obtained two-dimensional human pose to a high-dimensional space;

[0047] 2) extracting the local and global features of the key points through the Transformer, fusing the local and global features, sending the fused features into the space global Transformer encoder to extract the information in time sequence;

[0048] 3) The information in time sequence is sent into the single-layer perception machine decoder to obtain the final three-dimensional human posture key points.

[0049] In the above step 1), the estimation of the two-dimensional human posture according to the three-dimensional human posture estimation dataset comprises the following steps:

[0050] 1.1.1) Preprocess the three-dimensional human posture estimation dataset to obtain a picture sequence comprising two-dimensional human postures and corresponding three-dimensional human posture information;

[0051] 1.1.2) Send the two-dimensional human posture picture sequence into the two-dimensional human posture detector CPN to obtain a joint skeleton of the two-dimensional human posture, comprising a sequence of 17 two-dimensional human posture key points.

[0052] In the above step 1), the obtained two-dimensional human posture is lifted to a high-dimensional space, comprising the following steps:

[0053] 1.2.1) The obtained two-dimensional human posture sequence is equally divided into m sequences, wherein each sequence has f frames (inadequate 0 padding), and each frame is a two-dimensional space coordinate comprising 17 key points, that is, the skeleton sequence is divided into a matrix sequence Further Each x i is a two-dimensional human key point coordinate of each frame; wherein f is the frame number of the input skeleton, J is the number of key points, and 2 is the two-dimensional coordinate of the key point, wherein is a real number field space.

[0054] 1.2.2) The two-dimensional space point coordinates are lifted to a c-dimensional feature space and are encoded to obtain a feature matrix That is:

[0055] Y=[x 1 E;x 2 E;…;x f E]+E pos

[0056] Wherein, is a coordinate projection matrix for projecting two-dimensional coordinates to a c-dimensional space, E pos is a coordinate encoding matrix, because the Transformer encoder used later ignores position information, we set a set of learnable matrices E pos to positionally encode the features.

[0057] In the step 2), the local and global features of the key points are extracted by the Transformer, specifically: the transformed two-dimensional human skeleton sequence is segmented into an upper half limb sequence Jup and a lower half limb sequence Jdown, and Jup and Jdown are respectively sent to the local information encoder, and J is sent to the global spatial key point Transformer encoder; wherein J = Jup + Jdown.

[0058] In the embodiment, the obtained feature matrix Y is segmented into an upper local information matrix and a lower local information matrix wherein Jup + Jdown = J, wherein is a real number field space; by the feature point segmentation method, the local information can be better obtained, and the complex posture is decomposed into a combination of simple postures, then Jup and Jdown are respectively sent to the local spatial key point Transformer encoder, and J is sent to the global spatial key point Transformer encoder.

[0059] wherein the Transformer encoder includes a multi-head attention mechanism and a multi-layer perception machine, as follows:

[0060] Y l ′ = MSA (LN (Y l-1 ) ) + Y l-1

[0061] Y l = MLP (LN (Y l ′) ) + Y l ′

[0062] wherein l represents the number of layers of the Transformer encoder, in the embodiment, l = 1, 2, 3, 4, LN is a layer regularization method LayerNorm, MSA is a multi-head attention mechanism, MLP is a multi-layer perception machine, Y l ′ represents the result obtained by the first linear layer of the lth layer multi-layer perception machine, Y l represents the result obtained by the second linear layer of the lth layer multi-layer perception machine.

[0063] In the calculation of the multi-head attention mechanism, the attention of each head is calculated first, and the features of each head are fused to obtain the final multi-head attention;

[0064] After using the multi-head attention mechanism, we obtain the upper half spatiotemporal self-attention feature matrix composed of key points in the upper half such as arms, head and torso, and the lower half spatiotemporal self-attention feature matrix composed of key points in the lower half such as legs, feet and hips. The features of the two feature matrices are then fed into the regularization layer to regularize the data.

[0065] Then, through a multilayer sensing mechanism, the two spatiotemporal self-attention states are updated separately to obtain the spatiotemporal feature matrix F(Z). up ), and perform regularization.

[0066] In this embodiment, the attention of each head is:

[0067]

[0068] Where Q, K, and V are projections of the original features;

[0069] Q = YW Q K = YW k V = YW V

[0070] MSA(Q,K,V)=Concat(H1,H2,…,H h W out

[0071] where H i =Attention(Q) i ,K i V i ), i∈[1,…,h]

[0072] Where H i This represents the computational result of a single-layer self-attention mechanism, where h represents the number of multi-head attention mechanisms.

[0073] The calculation method for a multilayer perceptron is as follows:

[0074]

[0075] The above local information matrix Y up For example, first, the data is regularized, and then the attention matrix for different key points in each frame is calculated. For the key point feature matrix of each frame... The corresponding Q, K, and V are:

[0076] Q = Y up W Q K = Y up W k V = Y up W V

[0077] W Q W K , The features of each head are then fused to obtain the final multi-head attention:

[0078] MSA(Q,K,V)=Concat(H1,H2,…,H h W out

[0079] where H i =Attention(Q) i ,K i V i ), i∈[1,…,h]

[0080] After applying the multi-head attention mechanism, we obtain the spatiotemporal self-attention feature matrices of the human body in two dimensions: upper and lower. and This feature is then fed into a regularization layer to regularize the data. Following this, a multilayer perceptron is used. The multilayer perceptron mechanism is mainly described as follows: for the input...

[0081] F(Z up )=G(b (2) +W (2) (G(b (1) +W (1) Z up )))

[0082] in Let b be the weight matrix. (1) b (2) Let G be the paranoia vector, and G be the activation function Sigmoid. Then, the resulting spatiotemporal feature matrix F(Z) is... up Regularization is performed.

[0083] Step 2) above involves extracting timing information, including the following steps:

[0084] 2.1) Each frame obtained and The frames are fused together, and f frames within the adjacent range are taken to form a sequence.

[0085] 2.2) Transform the feature matrix to obtain the final global feature matrix containing time information. The feature matrix is ​​fed into a spatiotemporal encoder similar to the Transformer encoder described above to extract spatiotemporal features, resulting in a final matrix containing both time and global features.

[0086] In step 3) above, the information in time sequence is sent into a single-layer perception machine decoder to obtain final three-dimensional human posture key points, including the following steps:

[0087] 3.1) One-dimensional convolution is adopted to reduce f frame features to one frame to generate an intermediate frame key point feature matrix

[0088] 3.2) A single-layer perception machine is adopted as a decoder to reduce c-dimensional feature matrices in a feature space to three dimensions:

[0089] F(TZ)=G(b (1) +W (1) TZ)

[0090] Wherein, G is a sigmoid activation function, and after matrix transformation, final three-dimensional human postures are obtained

[0091] 3.3) The loss function L of the network is calculated:

[0092]

[0093] Wherein, p k is a real three-dimensional key point coordinate, is an estimated three-dimensional human posture coordinate, wherein J is the number of all human key points, and k is the kth key point. Finally, the error is back-propagated for training.

[0094] In summary, the application fuses local and global features, divides complex postures into simple postures, improves the detection accuracy of three-dimensional human postures, fuses time and space features, reduces the jitter of three-dimensional human postures between adjacent frames by adopting the method of obtaining attention between adjacent frames, and reduces the complexity of human posture acquisition without using camera calibration and other methods.

[0095] Embodiment:

[0096] The application examples will be described in detail below. Figure 2 、 Figure 3 .

[0097] 1) In the present example, the Human3.6M dataset is adopted, which is the most widely used dataset in the field of three-dimensional human posture estimation. The dataset contains 17 scenes performed by 11 professional actors, a total of 3.6 million pictures. The commonly used division method is to divide S1, S5, S7 and S8 as the training set, S9 and S11 as the test set, and the method of the application is also divided according to this method, and then the two-dimensional image and the corresponding three-dimensional real coordinate value are extracted;

[0098] 2) send the image into the trained two-dimensional human pose detector CPN to get the two-dimensional human skeleton sequence, and save the human skeleton sequence data;

[0099] 3) divide the input skeleton sequence into a matrix with a batch size of 1024, 81 frames of sampling, 17 key points, and 2 spatial coordinates, then send the matrix into a neural network linear layer with an input feature of 2 and an output feature of 32, and then add a learnable position coding matrix J with a size of 17x32 to each frame.

[0100] 4) divide the feature matrix containing coding information obtained in step 3) into an upper half sequence Jup with a size of (1024x81)x10x32 and a lower half sequence Jdown with a size of (1024x81)x7x32, and then send Jup, Jdown and J into the corresponding depth-4 Transformer coding layer respectively.

[0101] 5) fuse the two local feature matrices obtained in 4), then add them to the global feature matrix to synthesize a new feature matrix, then transform the (1024x81)x17x32 matrix into a 1024x81x(17x32) matrix for fusing the time information between different frames, then add an 81x(17x32) position coding matrix to the matrix and send it into the Transformer time domain information coding layer similar to that described in 3) to obtain the processed feature matrix.

[0102] 6) obtain the three-dimensional pose feature matrix 1024x1x(17x32) of the intermediate frame by one-dimensional convolution with an input dimension of 81 and an output dimension of 1, then obtain the final three-dimensional human pose 1024x1x17x3 by linear layer, and calculate the error with the true value label.

[0103] In the above step 4), 4 Transformer layers are included, each Transformer encoding layer includes a projection layer, a multi-head attention layer, a layer normalization layer and a multi-layer perceptron layer. The specific structure is that the 17x32 dimensional matrix is first projected to a feature matrix of the same dimension by the projection layer, that is, the Q, K and V matrices mentioned in the last section are obtained after the neural network linear layer, then the multi-head attention mechanism is calculated, in this embodiment, 8 heads are used, that is, 8 different self-attention mechanisms are used to capture features of different scales, and each attention mechanism is scaled to prevent the activation layer from being biased due to excessive dimension, then the attention mechanisms are fused to obtain a 17x32 feature matrix, finally the obtained result is sent to the layer normalization layer with a feature dimension of 32, then it is sent to the 2-layer perceptron layer, the first layer increases the 32-dimensional feature to 4x32 layers, then it is sent to the sigmoid activation layer and the dropout layer, then the 4x32 feature is reduced to 32 layers, then it is sent to the sigmoid activation layer and the dropout layer, wherein the features before entering the Transformer, the features after the multi-head attention and the features after the multi-layer perceptron are connected in residual connection to prevent gradient vanishing and gradient explosion.

[0104] In the above step 5), the input matrix is adjusted to 81x(17x32), which is to obtain the time information of 81 frames of samples, and the feature matrix after the multi-head attention mechanism is calculated is 81x(17x32), the hidden layer dimension of the multi-layer perceptron is 4x(17x32), and the dimension of each layer normalization is also changed to (17x32). In addition, the structure is similar to that in step 4).

[0105] To verify the effectiveness and practicability of the method, an example on Human3.6M dataset is given below, and Table 1 is the detection result of the example on the test set, and each type of measurement standard is MPJPE (Mean Per Joint Position Error), which is the average error value of all key points. The full name and abbreviation of the 15 action categories in the dataset are defined as: Directions (Dir), Discussion (Disc), Eating (Eat), Greeting (Greet), Talking on the phone (Phone), Taking photo (Photo), Posing (Pose), Making purchases (Purch), Sitting (Sit), Sitting Down (SitD), Smoking (Smoke), Waiting (Wait), Walking dog (WalkD), Walking (Walk), and Walking together (WalkT).

[0106] Table 1 The result of the method of the application evaluated by MPJPE error on Human3.6M dataset

[0107]

[0108] As shown in Table 1, compared with other monocular human pose estimation methods, the improved method based on spatio-temporal information fusion proposed by the application achieves better results, and most of the test categories of the method of the application have a larger improvement compared with other methods, which proves the effectiveness of the method of the application, andThe visualization result of the application also proves the effectiveness of the method of the application. Figure 4

[0109] In an embodiment of the application, a monocular three-dimensional human pose estimation system fusing spatio-temporal features is provided, which fuses local and global spatial information and combines time information, comprising:

[0110] A two-dimensional human pose estimation module estimates the two-dimensional human pose according to the three-dimensional human pose estimation dataset, and elevates the obtained two-dimensional human pose to a high-dimensional space;

[0111] A feature fusion module extracts local and global features of key points through a Transformer, fuses the local and global features, and sends the fused features into a spatial global Transformer encoder to extract information in time sequence;

[0112] The pose key point acquisition module sends temporal information into the single-layer perceptron decoder to obtain the final 3D human pose key points.

[0113] The system provided in this embodiment is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0114] like Figure 5 The diagram illustrates a computing device structure according to an embodiment of the present invention. This computing device can be a terminal and may include a processor, a communication interface, memory, a display screen, and an input device. The processor, communication interface, and memory communicate with each other via a communication bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program, which, when executed by the processor, implements an estimation method. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, a management network, NFC (Near Field Communication), or other technologies. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the casing of the computing device, or an external keyboard, touchpad, or mouse. The processor can call logical instructions in memory to execute the following method: estimate the two-dimensional human pose based on the three-dimensional human pose estimation dataset, and upscale the obtained two-dimensional human pose to a high-dimensional space; extract local and global features of key points through a Transformer, fuse the local and global features, and feed the fused features into a spatial global Transformer encoder to extract temporal information; and feed the temporal information into a single-layer perceptron decoder to obtain the final three-dimensional human pose key points.

[0115] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0116] Those skilled in the art can understand that, Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computing device to which the scheme of the present application is applied. The specific computing device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0117] In an embodiment of the present application, a computer program product is provided, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the method provided by the above-mentioned method embodiments, for example comprising: performing estimation of a two-dimensional human pose according to a three-dimensional human pose estimation dataset, and lifting the obtained two-dimensional human pose to a high-dimensional space; extracting local and global features of key points by a Transformer, fusing the local and global features, and sending the fused features to a spatial global Transformer encoder to extract information in time sequence; and sending the information in time sequence to a single-layer perception machine decoder to obtain final three-dimensional human pose key points.

[0118] In an embodiment of the present application, a non-transitory computer-readable storage medium is provided, which stores server instructions, the computer instructions causing a computer to execute the method provided by each of the above embodiments, for example comprising: performing estimation of a two-dimensional human pose according to a three-dimensional human pose estimation dataset, lifting the obtained two-dimensional human pose to a high-dimensional space; extracting local and global features of key points by a Transformer, fusing the local and global features, and sending the fused features into a spatial global Transformer encoder to extract information in time sequence; and sending the information in time sequence into a single-layer perceptron decoder to obtain a final three-dimensional human pose key point.

[0119] The computer-readable storage medium provided by the above embodiments has similar implementation principles and technical effects to the above method embodiments, and thus will not be described here.

[0120] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices generate a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0121] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0122] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0123] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A monocular 3D human pose estimation method fusing spatio-temporal features, characterized in that, The local and global spatial information is fused, and the time information is combined, including: According to the three-dimensional human posture estimation dataset, the estimation of the two-dimensional human posture is performed, and the obtained two-dimensional human posture is lifted to a high-dimensional space, specifically including: the obtained two-dimensional human posture sequence is equally divided into m sequences, wherein each sequence has f frames, and each frame is a two-dimensional space coordinate containing 17 key points, that is, the skeleton sequence is divided into matrix sequence, and then each of the matrix sequence is a two-dimensional human key point coordinate of each frame; wherein J is the number of key points, and 2 is the two-dimensional coordinate of the key point. Lifting two-dimensional spatial point coordinates to c-dimensional feature space and performing coordinate coding to obtain a feature matrix ; The local and global features of the key points are extracted by the Transformer, the local and global features are fused, the fused features are sent into a global time Transformer encoder, information in time sequence is extracted, and a final matrix containing time and global features is obtained The local and global features of the key points are extracted by the Transformer, the local and global features are fused, the fused features are sent into a global time Transformer encoder, information in time sequence is extracted, and a final matrix containing time and global features is obtained Specifically: The obtained feature matrix Y is segmented into an upper local information matrix and a lower local information matrix , wherein is a real number field space; local information is obtained by a feature point segmentation method, a complex pose is decomposed into a combination of simple poses, and then the upper half body sequence Jup and the lower half body sequence Jdown are respectively sent into a local spatial key point Transformer encoder, and J is sent into a global spatial key point Transformer encoder; The information on the time sequence is sent into a single-layer perception machine decoder to obtain final three-dimensional human posture key points, including: reducing the f-frame features to one frame by one-dimensional convolution to generate the key point feature matrix of the intermediate frame ; The single-layer perception machine is used as the decoder to reduce the c-dimensional feature matrix of the feature space to three dimensions: wherein, , G is sigmoid activation function, after matrix transformation, get the final three-dimensional human pose ; The loss function L of the network is calculated: wherein, are real three-dimensional keypoint coordinates, are estimated three-dimensional human pose coordinates, wherein J is the number of all human key points, and k is the kth key point.

2. The monocular 3D human pose estimation method fusing spatio-temporal features according to claim 1, wherein, The two-dimensional human posture estimation according to the three-dimensional human posture estimation dataset includes: The three-dimensional human posture estimation dataset is preprocessed to obtain a picture sequence including a two-dimensional human posture and corresponding three-dimensional human posture information; The two-dimensional human posture picture sequence is sent into a two-dimensional human posture detector to obtain a two-dimensional human posture key node skeleton.

3. The monocular 3D human pose estimation method fusing spatio-temporal features of claim 1, wherein, The Transformer encoder includes a multi-head attention mechanism and a multi-layer perception machine; In the calculation of the multi-head attention mechanism, the attention of each head is calculated, and the features of each head are fused to obtain the final multi-head attention; After the multi-head attention mechanism, the upper half of the space-time self-attention feature matrix and the lower half of the space-time self-attention feature matrix are obtained, and the features of the upper and lower feature matrices are sent into a regularization layer to regularize the data; The obtained two parts of spatio-temporal self-attention states are separately updated by a multi-layer perception mechanism to obtain a spatio-temporal feature matrix and F(Z down ) and are regularized.

4. The monocular 3D human pose estimation method fusing spatio-temporal features according to claim 3, wherein, The information on the time sequence includes: The obtained each frame and is fused, and the frames in the adjacent range constitute a sequence; Transform the feature matrix to get the final global feature matrix containing time information , the feature matrix is input into a space-time encoder similar to the above Transformer encoder to extract space-time features, and the final matrix containing time and global features is obtained .

5. A system for monocular 3D human pose estimation fusing spatio-temporal features, for implementing the method for monocular 3D human pose estimation fusing spatio-temporal features according to any one of claims 1-4, characterized in that, The local and global spatial information is fused, and the time information is combined, including: The two-dimensional human posture estimation module estimates the two-dimensional human posture according to the three-dimensional human posture estimation dataset, and the obtained two-dimensional human posture is lifted to a high-dimensional space; The feature fusion module fuses the local and global features of the key points by the Transformer, fuses the local and global features, and sends the fused features into a global time Transformer encoder to extract information on the time sequence; The posture key point acquisition module sends the information on the time sequence into a single-layer perception machine decoder to obtain final three-dimensional human posture key points.

6. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that when executed by a computer cause the computer to perform a method comprising: The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods of claims 1-4.

7. A computing device, comprising: Including: One or more processors, memories, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods of claims 1-4.

Images