A method, apparatus, and storage medium for estimating 3D human pose.

Abstract

This invention provides a method, apparatus, and storage medium for estimating 3D human pose, comprising: S1, inputting a continuous sequence of two-dimensional pose frames S into a position embedding module to obtain position features F. p S2, the inter-frame Transformer module utilizes the Transformer structure and positional features F p The system learns spatial location information, calculates inter-frame attention features and offset attention features to learn the inter-frame correlation of each joint in sequence S, and obtains the predicted 3D pose sequence P through a fully connected layer. s S3, the initial 3D joint positions of each frame are sequentially input into the intra-frame Transformer module. By calculating the self-attention between joints in each frame, the joint relationships in each frame are learned, and the optimized 3D joint positions of the target frame are obtained. This technical solution comprehensively perceives the intra-frame human joint relationships and inter-frame joint correlations, thereby outputting accurate 3D human pose.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method, apparatus and storage medium for estimating 3D human pose. Background Technology

[0002] Currently, 3D pose estimation methods have limitations in handling issues such as mutual occlusion, unique poses, and cluttered backgrounds in the data. Existing 3D pose estimation methods are mainly divided into top-down and bottom-up methods. Top-down methods typically first use object detection techniques to infer 2D pose, and then perform a second stage of 3D human pose estimation. Therefore, this method is affected by the accuracy of object detection, has a disadvantage in handling occlusion problems, and the two-stage processing increases computational resources. Bottom-up estimation methods are fast, but they extract insufficient keypoint features, which is detrimental to the accuracy of the estimation. Summary of the Invention

[0003] To address the aforementioned problems in the prior art, embodiments of the present invention provide a method, apparatus, and storage medium for estimating 3D human pose, which eliminates the need for object detection to infer 2D pose and allows for focus on feature extraction of key points.

[0004] To achieve the above objectives, on the one hand, a method for estimating 3D human pose is provided, including:

[0005] S1, input the two-dimensional attitude continuous frame sequence S into the position embedding module to obtain position features F. p The sequence S = {f1, f2, f3, ..., f...} is given by the given sequence S = {f1, f2, f3, ..., f...}. N}, f i ∈R J×d Let J represent the two-dimensional position of the human joint in the i-th frame, J represent the number of joints, d represent the position dimension, and N represent the number of frames contained in the sequence S. The position feature is the position feature of the human joint.

[0006] S2, the position feature F p The sequence S is input to the inter-frame Transformer module, which utilizes the Transformer structure and the positional feature F. p By learning spatial location information, calculating attention features and offset attention features between frames, and thus learning the inter-frame correlation of each joint in sequence S, the predicted 3D pose sequence P is obtained through a fully connected layer. s , where P s ∈R N×J×3 The predicted 3D pose sequence P sIt contains the initial 3D joint positions for each frame in sequence S;

[0007] S3, the initial 3D joint positions of each frame are sequentially input into the intra-frame Transformer module, and the joint relationships in each frame are learned by calculating the self-attention between joints in each frame, and the optimized target frame 3D joint positions are obtained.

[0008] Preferably, in the estimation method, step S2 includes the following steps performed by the inter-frame Transformer module:

[0009] S21, perform a first linear transformation, a second linear transformation, and a third linear transformation on the sequence S input to the inter-frame Transformer module to obtain the query vector matrix Q of the inter-frame Transformer module. q , key vector matrix Q k Sum value vector matrix Q v ;

[0010] S22, regarding the position feature F p The query vector matrix Q q The key vector matrix Q k and the value vector matrix Q v Encode the data and calculate the positional deviation P. bias , where P bias =Q q ·F P +Q k ·F P +Q v ·F P ;

[0011] S23, calculate the attention weight W′ on the query vector matrix and the key vector matrix through a dot product operation. A , where W′ A =Q q ·Q k T +P bias ;

[0012] S24, regarding the W′ A Perform a SoftMax operation to obtain the normalized weights W. A , where W A =SoftMax(W′ A );

[0013] S25, Calculate attention feature F A F A =W A ·Q v ;

[0014] S26, Calculate the offset attention feature F offset , of which F offset =F A -F P ;

[0015] S27, Perform a multilayer perceptron operation on the offset attention features to obtain the features F extracted by the multilayer perceptron. OM , of which F OM =MLP(F offset );

[0016] S28, Calculate the output feature F of the inter-frame Transformer module. out , of which F out =F A +F OM .

[0017] Preferably, in the estimation method, step S2 further includes:

[0018] The steps S21 to S28 are repeated iteratively a predetermined number of times M, and the output feature F obtained each time is... out Let them be F1, F2, ... F M , where M is an integer greater than 1;

[0019] Specifically, when step S21 is executed for the first time, the input of the inter-frame Transformer module is the sequence S; from the second time onwards, the input of the inter-frame Transformer module is the output feature of the previous step of the inter-frame Transformer module, and step S21 is to perform a first linear transformation, a second linear transformation, and a third linear transformation on the previous output feature input to the inter-frame Transformer module.

[0020] The F1, F2, ... F M By concatenating them together using the Contat operation, we obtain the global feature F. G , of which F G =Contat(F1, F2, ..., F) M );

[0021] Apply the fully connected layer to the F G The predicted 3D pose sequence P is obtained. s .

[0022] Preferably, in the estimation method, wherein,

[0023] In step S1, the position feature F P=Relu(Norm(Conv(S))), F P ∈R N×J×d′ d′ represents the feature dimension, Conv represents the convolution operation, Norm represents the normalization operation, and Relu represents the Relu activation function;

[0024] In step S2, steps S21 to S28 are repeated iterated 4 times.

[0025] In step S2, the three fully connected layers are applied to the F G .

[0026] Preferably, in the estimation method, step S3 includes:

[0027] S31, input the initial 3D joint position p of the single-frame target frame into the intra-frame Transformer module, where p = {x1, x2, x3, ..., x...} J}, x1, x2, x3, ..., x J This represents the 3D joint position of each individual joint in the input target frame, where x J This represents the 3D joint position of the J-th joint in the input target frame, where J represents the number of joints;

[0028] S32, the intra-frame Transformer module performs a fourth linear transformation, a fifth linear transformation, and a sixth linear transformation on p respectively to obtain the query vector matrix Q′ of the intra-frame Transformer module. q , key vector matrix Q′ k Sum value vector matrix Q′ v ;

[0029] S33, the intra-frame Transformer module calculates the attention weight w′ through a dot product operation. A , where w′ A =Q′ q ·Q′ k T ;

[0030] S34, the intra-frame Transformer module adjusts the attention weight w′ A Perform a SoftMax operation to obtain the normalized weights w. A , where w A =SoftMax(w′) A );

[0031] S35, the intra-frame Transformer module calculates the attention feature f A , where f A =wA ·Q′ v ,

[0032] Furthermore, the output feature of the intra-transformer layer of the intra-transformer module is f out =f A .

[0033] Preferably, in the estimation method, step S3 further includes:

[0034] The steps S31 to S35 are repeated iterated a predetermined number of times K, and the output feature f is obtained each time. out Represented as f out1 , ...f outk , where K is an integer greater than 1;

[0035] Specifically, when step S31 is executed for the first time, the initial 3D joint position p of the single-frame target frame is input to the intra-frame Transformer module. From the second time onwards, the output features of the intra-frame Transformer module in the previous step are input to the intra-frame Transformer module.

[0036] f out1 , ...f outk The local feature F is obtained by concatenating the elements through a superposition operation. l ;

[0037] Apply a fully connected layer to the local feature F l The optimized 3D joint positions of the target frame are obtained.

[0038] Preferably, in the estimation method, step S3 involves repeating steps S31 to S35 twice.

[0039] Preferably, in the estimation method, the inter-frame Transformer module and the intra-frame Transformer module are trained inter-frame Transformer modules and intra-frame Transformer modules, wherein the training of the inter-frame Transformer module and the intra-frame Transformer module is jointly supervised using multiple loss functions.

[0040] The expression for the combination of multiple loss functions is as follows: Where ω is an adjustable threshold, and its value ranges from (0, 1);

[0041] in, Let be the first loss function, and .

[0042]

[0043] in, Let J be the loss function for the i-th joint, and g be the number of joints. i p represents the actual 3D joint position of the i-th joint. i σ represents the 3D joint position of the i-th joint estimated using the inter-frame Transformer module and the intra-frame Transformer module during training, where σ is an adjustable threshold.

[0044] in Let be the second loss function, and

[0045]

[0046] On the other hand, an apparatus for estimating 3D human pose is provided, including a memory and a processor, the memory storing at least one program, the at least one program being executed by the processor to implement any of the methods described above.

[0047] In another aspect, a computer-readable storage medium is provided, wherein at least one program is stored therein, the at least one program being executed by a processor to implement any of the methods described above.

[0048] The above technical solution has the following technical effects:

[0049] The technical solution of this invention obtains positional features through a position embedding structure, and then feeds these features into an inter-frame Transformer module to learn spatial positional information. This inter-frame Transformer module calculates attention features and offset attention features between frames based on the positional features and the frame sequence, focusing on learning the global correlation of each joint. After determining the predicted 3D joint positions using the acquired global features, the technical solution of this invention further utilizes an intra-frame Transformer module to calculate self-attention between joints in each frame, learning the body joint relationships in each frame. Thus, the technical solution of this invention, based on a Transformer structure jointly optimized for inter-frame and intra-frame connections, achieves comprehensive perception of intra-frame human joint relationships and inter-frame correlations, thereby enabling the output of accurate 3D human pose.

[0050] Therefore, the 3D human pose estimation method based on inter-frame and intra-frame Transformer in this embodiment of the invention overcomes the shortcomings of existing top-down and bottom-up 3D human pose estimation methods, does not require target detection technology to infer 2D pose and can focus on feature extraction of key points. Attached Figure Description

[0051] Figure 1This is a flowchart illustrating a 3D human pose estimation method according to an embodiment of the present invention.

[0052] Figure 2 This is a flowchart illustrating a 3D human pose estimation method according to another embodiment of the present invention.

[0053] Figure 3 This is a schematic diagram of the structure of a 3D human posture estimation device according to an embodiment of the present invention. Detailed Implementation

[0054] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments and to explain the operating principles of the embodiments in conjunction with the relevant descriptions in the specification. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention. Components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0055] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0056] Example 1:

[0057] Figure 1 This is a flowchart illustrating a 3D human pose estimation method according to an embodiment of the present invention. Figure 1 The 3D human pose estimation method in this embodiment includes:

[0058] S1, input the two-dimensional attitude continuous frame sequence S into the position embedding module to obtain position features F. p Where the sequence S = {f1, f2, f3, ..., f...} N}, f i ∈R J×d Let J represent the two-dimensional position of the human joint in the i-th frame, where J represents the number of joints, d represents the position dimension, N represents the number of frames in the sequence S, and the position feature is the position feature of the human joint.

[0059] S2, the position feature F p The sequence S is input to the inter-frame Transformer module, which utilizes the Transformer structure and the positional feature F. p By learning spatial location information, calculating attention features and offset attention features between frames, and thus learning the inter-frame correlation of each joint in sequence S, the predicted 3D pose sequence P is obtained through a fully connected layer. s , where P s ∈R N×J×3 The predicted 3D pose sequence P sIt contains the initial 3D joint positions for each frame in sequence S;

[0060] S3 sequentially inputs the initial 3D joint positions of each frame into the intra-frame Transformer module. By calculating the self-attention between joints in each frame, the module learns the joint relationships in each frame and obtains the optimized 3D joint positions of the target frame. Once the 3D joint positions are determined, the 3D human pose is determined accordingly.

[0061] In this embodiment, the position embedding module can be implemented using existing position embedding structures, models, or networks. For example, the position features F obtained through position embedding operations... P =Relu(Norm(Conv(S))), F P ∈R N×J×d′ d′ represents the feature dimension, Conv represents the convolution operation, Norm represents the normalization operation, and Relu represents the Relu activation function.

[0062] In this embodiment, the inter-frame Transformer module is a Transformer structure used between frames, and the intra-frame Transformer module is a Transformer structure used within a frame. The 3D human pose estimation method provided by this embodiment utilizes a Transformer structure jointly optimized for inter-frame and intra-frame use to output accurate 3D human pose by comprehensively perceiving the human joint relationships within frames and the correlations between frames.

[0063] Preferably, before formally using the inter-frame Transformer module and the intra-frame Transformer module for actual 3D human pose estimation, it is necessary to first train the inter-frame Transformer module and the intra-frame Transformer module using a 2D pose frame dataset. Multiple loss functions are used to jointly supervise the training of the modules or model to minimize the error between the estimated pose and the true pose. Specific multiple loss functions are described below.

[0064] Example 2:

[0065] Figure 2 This is a flowchart illustrating a 3D human pose estimation method according to another embodiment of the present invention; as shown below. Figure 2 The 3D human pose estimation method of this embodiment includes the following steps:

[0066] S101, Input a two-dimensional attitude continuous frame sequence S, where S = {f1, f2, f3, ..., f...} N}, where f i ∈R J×d, represents the two-dimensional position of the human joint in the i-th frame, J represents the number of joints, d represents the position dimension, and N represents the number of frames contained in the sequence S;

[0067] This step involves inputting a continuous two-dimensional pose frame sequence S into the entire model structure used to implement the 3D human pose estimation method of this invention. The entire model structure may include: a location embedding module or structure, an inter-frame Transformer module or structure, an intra-frame Transformer module or structure, and a fully connected layer.

[0068] S102, feed the above continuous frame sequence S into the location embedding structure or module to obtain location features F. p In one implementation, the positional feature F P =Relu(Norm(Conv(S))), F P ∈R N×J×d′ d′ represents the feature dimension, Conv represents the convolution operation, Norm represents the normalization operation, and Relu represents the Relu activation function;

[0069] S103, input the original two-dimensional attitude continuous frame sequence S into the inter-frame Transformer module or structure;

[0070] S104, the inter-frame Transformer module performs a linear transformation on the continuous two-dimensional pose frame sequence S, i.e., the first linear transformation, to obtain the weight matrix Q. q As a query vector matrix, it is represented as Q. q =S·W q W q This represents the linear transformation matrix used to obtain the query vector matrix;

[0071] S105, the inter-frame Transformer module performs a linear transformation (the second linear transformation) on the continuous two-dimensional pose frame sequence S to obtain the weight matrix Q. k As a key vector matrix, it is represented as Q. k =S·W k W k This represents the linear transformation matrix used to obtain the key vector matrix;

[0072] S106, the inter-frame Transformer module performs a linear transformation (the third linear transformation) on the continuous two-dimensional pose frame sequence S to obtain the weight matrix Q. v As a value vector matrix, it is represented as Q. v =S·W v W v This represents the linear transformation matrix used to obtain the value vector matrix;

[0073] For example, the above linear transformation matrices can be selected and set according to requirements;

[0074] S107, regarding location feature F p Query matrix Q q key matrix Q k and value matrix Q v Encode the data and calculate the positional deviation P. bias , where P bias =Q q ·F P +Q k ·F P +Q V ·F P ;

[0075] S108, perform a dot product operation on the query matrix Q. q and key matrix Q k Calculate attention weights W′ A , where W′ A =Q q ·Q k T +P bias ;

[0076] S109, regarding the attention weight W′ A Perform a SoftMax operation to obtain the normalized attention weights W. A , where W A =SoftMax(W′ A );

[0077] S110, the normalized attention weights W A With the value matrix Q v Matrix multiplication is performed to obtain the attention feature F. A , of which F A =W A ·Q v .

[0078] S111, through positional feature F p and attention feature F A Calculate the offset attention feature F offset , of which F offset =F A -F P ;

[0079] S112, regarding the offset attention feature F offset Perform multilayer perceptron operations to obtain the features F extracted by the multilayer perceptron. OM , of which F OM=MLP(F offset );

[0080] S113, calculate the output feature F of the inter-frame Transformer layer, i.e., the inter-frame Transformer module. out , of which F out =F A +F OM ;

[0081] S114, repeating the operations S103 to S112 a predetermined number of times, preferably 4 times, to obtain the output feature F. out These are denoted as F1, F2, F3, and F4, respectively. The first input to the inter-frame Transformer module is a two-dimensional pose continuous frame sequence S, and the input to the inter-frame Transformer module in subsequent iterations is the output feature of the previous iteration.

[0082] S115, F1, F2, F3, and F4 are concatenated together using a Concat operation to obtain the global feature F of the input frame sequence. G F G =Concat(F1, F2, F3, F4);

[0083] S116, Apply the fully connected layers, preferably three fully connected layers, to the global feature F. G The predicted 3D pose sequence P is obtained. s This yields the initial 3D joint positions in each target frame of sequence S;

[0084] S117, the initial 3D joint positions of the target frame obtained above are optimized by setting an intra-frame Transformer module or structure; the intra-frame Transformer module or structure is in one target frame at a time, i.e., a single-frame target frame, to finally obtain the 3D joint positions of each joint in the target frame; specifically, each time the initial 3D joint position p of the single-frame target frame obtained by the aforementioned inter-frame Transformer module is input, where p = {x1, x2, x3, ..., x...} J}, x1, x2, x3, ..., x J This represents the 3D joint position of each individual joint in the input target frame, where x1 is the 3D joint position of the selected first joint, x2 is the 3D joint position of the selected second joint, and x... J This represents the 3D joint position of the J-th joint in the input target frame, where J represents the number of joints;

[0085] S118, the intra-frame Transformer module performs a linear transformation (the fourth linear transformation) on the initial 3D joint position p of the single-frame target frame to obtain the weight matrix Q′.q Q′ q As a query vector matrix for an intra-frame Transformer module or structure, it is represented as Q′ q =p·W q′ ;, where W q′ This indicates obtaining Q′ q The linear transformation matrix used;

[0086] S119: The intra-frame Transformer module performs a linear transformation on the initial 3D joint position p of the above single-frame target frame, i.e., the fifth linear transformation, to obtain the weight matrix Q′. k As the key vector matrix of an intra-frame Transformer module or structure, it is represented as Q′. k =p·W k′ , where W k′ Show how to obtain Q′ k The linear transformation matrix used;

[0087] S120, the intra-frame Transformer module performs a linear transformation on the initial 3D joint position p of the aforementioned single-frame target frame, i.e., the sixth linear transformation, to obtain the weight matrix Q′. v As a value vector matrix, it is represented as Q′ v =p·W v′ , where W v′ Show how to obtain Q′ v The linear transformation matrix used;

[0088] Wherein, the linear transformation matrix W q′ 、W k′ and W v′ Select or set according to actual needs;

[0089] S121, the intra-frame Transformer module performs a dot product operation on the query matrix Q′. q and key matrix Q′ k Calculate attention weights w′ A , where w′ A =Q′ q ·Q′ k T ;

[0090] S122, Intra-frame Transformer module's attention weights w′ A Perform a SoftMax operation to obtain the normalized weights w A , represented as w A =SoftMax(W′ A );

[0091] S123: The normalized attention weights w of the intra-frame Transformer module A With the value matrix Q′ v Perform matrix multiplication to obtain the attention feature f. A , represented as f A =w A ·Q′ v The output feature of the intra-frame Transformer layer or module is f out =f A ;

[0092] S124, repeat operations S117 to S123 a predetermined number of times, preferably 2 iterations, to obtain the output feature f. out Let these be f1 and f2, respectively; where the first input of the intra-frame Transformer module is the initial 3D joint position p of the target frame mentioned above, and the input in subsequent iterations is the previous output feature f of the intra-frame Transformer module. out ;

[0093] S125, f1 and f2 are concatenated together through a superposition operation to obtain local features, represented as follows:

[0094] S126, apply a fully connected layer to local feature F l The optimized 3D joint positions of the target frames are obtained; after the N target frames in sequence S are optimized by the intra-frame Transformer module, the optimized 3D joint positions of each target frame constitute the optimized 3D pose sequence P∈R. N×J×3 .

[0095] In embodiments of the present invention, the neural network or model comprising inter-frame Transformer modules or structures and intra-frame Transformer modules or structures needs to be trained first. After training, actual 3D human pose estimation can be performed by executing the above steps S101 to S126. During model training, a continuous sequence of two-dimensional pose frames is used as the training dataset to train the model by executing the above steps S101 to S126, and multiple loss functions are used to optimize the trained model.

[0096] For example, the training of the model can be jointly supervised using two loss functions through the following steps:

[0097] S127, using the designed first loss function To optimize network training, and,

[0098]

[0099] in, Let J be the loss function for the i-th joint, and g be the number of joints. i p represents the actual 3D joint position of the i-th joint. i σ represents the 3D joint position of the i-th joint estimated using the inter-frame Transformer module and intra-frame Transformer module mentioned above during training, where σ is an adjustable threshold.

[0100] S128, using the designed second loss function Optimize network training, among which,

[0101]

[0102] S129 employs a joint supervision network with multiple loss functions, where the expression for the joint loss functions is: Wherein, ω is an adjustable threshold, and the value of ω ranges from (0, 1).

[0103] Example 3:

[0104] The present invention also provides a device for estimating 3D human pose, such as... Figure 3 As shown, the device includes a processor 301, a memory 302, a bus 303, and a computer program stored in the memory 302 and executable on the processor 301. The processor 301 includes one or more processing cores. The memory 302 is connected to the processor 301 via the bus 303. The memory 302 is used to store program instructions. When the processor executes the computer program, it implements the steps in the above-described method embodiment of Embodiment 1 of the present invention.

[0105] Furthermore, as an executable solution, the device for estimating the 3D human pose can be a computer unit, which can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer unit may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described structure of the computer unit is merely an example and does not constitute a limitation on the computer unit. It may include more or fewer components, or combine certain components, or use different components. For example, the computer unit may also include input / output devices, network access devices, buses, etc., and this embodiment of the invention does not limit this.

[0106] Furthermore, as an executable solution, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, connecting various parts of the entire computer unit via various interfaces and lines.

[0107] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer unit by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0108] Example 4:

[0109] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the embodiments of the present invention.

[0110] If the modules / units integrated in the computer unit are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0111] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A method for estimating 3D human pose, characterized in that, include: S1, input the two-dimensional attitude continuous frame sequence S into the position embedding module to obtain position features Fp, where the sequence S = {f1, f2, f3, ..., f...} N }, f i ∈R J×d Let J represent the two-dimensional position of the human joint in the i-th frame, J represent the number of joints, d represent the position dimension, and N represent the number of frames contained in the sequence S. The position feature is the position feature of the human joint. S2, the position feature F p The sequence S is input to the inter-frame Transformer module, which utilizes the Transformer structure and the positional feature F. p By learning spatial location information, calculating attention features and offset attention features between frames, and thus learning the inter-frame correlation of each joint in sequence S, the predicted 3D pose sequence P is obtained through a fully connected layer. s , where P s ∈R N×J×3 The predicted 3D pose sequence P s It contains the initial 3D joint positions for each frame in sequence S; S3, the initial 3D joint positions of each frame are sequentially input into the intra-frame Transformer module, and the joint relationships in each frame are learned by calculating the self-attention between joints in each frame, and the optimized target frame 3D joint positions are obtained. Step S2 includes the following steps performed by the inter-frame Transformer module: S21, perform a first linear transformation, a second linear transformation, and a third linear transformation on the sequence S input to the inter-frame Transformer module to obtain the query vector matrix Q of the inter-frame Transformer module. q , key vector matrix Q k Sum value vector matrix Q v ; S22, regarding the position feature F p The query vector matrix Q q The key vector matrix Q k and the value vector matrix Q v Encode the data and calculate the positional deviation P. bias , where P bias =Q q ·F P +Q k ·F P +Q v ·F P ; S23, calculate the attention weight W′ on the query vector matrix and the key vector matrix through a dot product operation. A , where W′ A =Q q ·Q k T +P bias ; S24, regarding the W′ A Perform a SoftMax operation to obtain the normalized weights W. A , where W A =SoftMax(W′ A ); S25, Calculate attention feature F A F A =W A ·Q v ; S26, Calculate the offset attention feature F offset , of which F offset =F A -F P ; S27, Perform a multilayer perceptron operation on the offset attention features to obtain the features F extracted by the multilayer perceptron. OM , of which F OM =MLP(F offset ); S28, Calculate the output feature F of the inter-frame Transformer module. out , of which F out =F A +F OM ; Step S2 further includes repeating steps S21 to S28 for a predetermined number of iterations M, with each iteration yielding an output feature F. out Let them be F1, F2, ... F M , where M is an integer greater than 1; Specifically, when step S21 is executed for the first time, the input of the inter-frame Transformer module is the sequence S; from the second time onwards, the input of the inter-frame Transformer module is the output feature of the previous step of the inter-frame Transformer module, and step S21 is to perform a first linear transformation, a second linear transformation, and a third linear transformation on the previous output feature input to the inter-frame Transformer module. The F1, F2, ... F M By concatenating them together using the Contat operation, we obtain the global feature F. G , of which F G =Contat(F1, F2, ..., F) M ); Apply the fully connected layer to the F G The predicted 3D pose sequence P is obtained. s .

2. The estimation method according to claim 1, characterized in that, In step S1, the position feature F P =Relu(Norm(Conv(S))), F P ∈R N×J×d′ d′ represents the feature dimension, Conv represents the convolution operation, Norm represents the normalization operation, and Relu represents the Relu activation function; In step S2, steps S21 to S28 are repeated iterated 4 times. In step S2, the three fully connected layers are applied to the F G .

3. The estimation method according to claim 1, characterized in that, The step S3 comprises: S31, input the initial 3D joint position p of the single-frame target frame into the intra-frame Transformer module, where p = {x1, x2, x3, ..., x...} J }, x1, x2, x3, ..., x J This represents the 3D joint position of each individual joint in the input target frame, where x J This represents the 3D joint position of the J-th joint in the input target frame, where J represents the number of joints; S32, the intra-frame Transformer module performs a fourth linear transformation, a fifth linear transformation, and a sixth linear transformation on p respectively to obtain the query vector matrix Q′ of the intra-frame Transformer module. q , key vector matrix Q′ k Sum value vector matrix Q′ v ; S33, the intra-frame Transformer module calculates the attention weight w′ through a dot product operation. A ,in S34, the intra-frame Transformer module adjusts the attention weight w′ A Perform a SoftMax operation to obtain the normalized weights w. A , where w A =SoftMax(w′) A ); S35, the intra-frame Transformer module calculates the attention feature f A , where f A =w A ·Q′ v , Furthermore, the output feature of the intra-transformer layer of the intra-transformer module is f out =f A .

4. The estimation method according to claim 3, characterized in that, Step S3 further includes: The steps S31 to S35 are repeated iterated a predetermined number of times K, and the output feature f is obtained each time. out Represented as f out1 , ...f outk , where K is an integer greater than 1; Specifically, when step S31 is executed for the first time, the initial 3D joint position p of the single-frame target frame is input to the intra-frame Transformer module. From the second time onwards, the output features of the intra-frame Transformer module in the previous step are input to the intra-frame Transformer module. f out1 , ...f outk The local feature F is obtained by concatenating the elements through a superposition operation. l ; Apply a fully connected layer to the local feature F l The optimized 3D joint positions of the target frame are obtained.

5. The estimation method according to claim 4, characterized in that, In step S3, steps S31 to S35 are repeated iterated twice.

6. The estimation method according to claim 1, characterized in that, The inter-frame Transformer module and the intra-frame Transformer module are trained inter-frame Transformer modules and intra-frame Transformer modules, wherein the training of the inter-frame Transformer module and the intra-frame Transformer module is jointly supervised using multiple loss functions. The expression for the combination of multiple loss functions is as follows: Where ω is an adjustable threshold, and its value ranges from (0, 1); in, Let be the first loss function, and . Among them, l i Let J be the loss function for the i-th joint, and g be the number of joints. i p represents the actual 3D joint position of the i-th joint. i σ represents the 3D joint position of the i-th joint estimated using the inter-frame Transformer module and the intra-frame Transformer module during training, where σ is an adjustable threshold. in Let be the second loss function, and 7. A device for estimating 3D human pose, characterized in that, It includes a memory and a processor, the memory storing at least one program, the at least one program being executed by the processor to implement the method as claimed in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is executed by a processor to implement the method as described in any one of claims 1 to 6.

Images