A method and system for three-dimensional human pose estimation

Abstract

The application discloses a three-dimensional human posture estimation method and system, relates to the technical field of computer vision, and comprises the following steps: extracting two-dimensional human posture key points from a monocular video picture sequence, and generating a two-dimensional human posture key point sequence; projecting the two-dimensional key point sequence to a feature space through nonlinear high-dimensional mapping, and generating a high-dimensional feature space matrix; inputting the high-dimensional feature matrix into a three-dimensional human posture recognition model which fuses motion constraints and frequency division space-time features, obtaining a three-dimensional human posture key point sequence, and realizing three-dimensional human posture estimation through three-dimensional coordinates. The method improves the robustness and detection precision of the monocular three-dimensional human posture estimation method. Error values are calculated for the relative motion speed, bone length and bone direction of the key points, so that the training is easier to converge and the training process is more stable.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a three-dimensional human pose estimation method and system. BACKGROUND

[0002] With the continuous evolution of video monitoring technology and artificial intelligence technology, the practical application demand of human three-dimensional pose automatic analysis and recognition technology based on video images is growing. Such demand shows important value in many fields such as road traffic safety, motion recognition, virtual reality, human-computer interaction, robot operation, specific place monitoring and dangerous place worker pose recognition. The currently widely deployed monitoring camera can only capture a single view of two-dimensional plane image. Researching how to realize human three-dimensional pose estimation method from the two-dimensional image obtained in the actual application scene has become a key problem to be solved. Compared with two-dimensional pose estimation, three-dimensional pose estimation of a single frame image is more challenging, because it needs to accurately restore the complete three-dimensional position of each joint from the blurred and noisy two-dimensional image, and different three-dimensional poses may project the same or similar two-dimensional form. The task of inferring three-dimensional pose only relying on single two-dimensional image is extremely difficult in technology.

[0003] Currently, the mainstream method in the field of human three-dimensional pose estimation can be mainly divided into single-step method and two-step method. Among them, the single-step method directly regresses the three-dimensional coordinates of each joint from the two-dimensional image through an end-to-end model without introducing intermediate representation or additional processing steps. The advantage of this kind of method is that the model architecture is relatively simple, but it is limited by the lack of intermediate constraint mechanism and the scarcity of high-quality three-dimensional pose labeling data, and it is difficult to directly infer three-dimensional space information from two-dimensional projection. In addition, this kind of method has high demand for computing resources, and needs to be optimized through fine hyperparameters to realize stable convergence.

[0004] On the other hand, the two-step method adopts a phased processing strategy: first, the two-dimensional key point detection model is used to extract the two-dimensional coordinates of the joints in the image, and then the two-dimensional to three-dimensional mapping relationship is constructed to realize three-dimensional pose reconstruction. Some methods rely on pre-constructed two-dimensional-three-dimensional pose corresponding library, and output three-dimensional results through nearest neighbor search or parameterized matching. However, this kind of method based on large-scale dictionary learning has obvious defects, which needs to consume a lot of time to construct high-dimensional mapping space in the training stage, and the inference stage needs to traverse the candidate pose library, which limits the real-time performance and is difficult to meet the low delay demand in dynamic scene. SUMMARY

[0005] In view of the above existing problems, the present application is proposed.

[0006] Therefore, this invention provides a three-dimensional human pose estimation method to address the problems of existing technologies lacking the ability to extract time-series feature information, requiring further improvement in the accuracy of three-dimensional human pose estimation, lacking a method to infer three-dimensional human pose estimation from continuous two-dimensional images, and having weak robustness in the designed related models.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides a three-dimensional human pose estimation method, which includes extracting two-dimensional human pose key points from a monocular video image sequence and generating a two-dimensional human pose key point sequence.

[0009] The two-dimensional human posture key point sequence is projected onto the feature space through a nonlinear high-dimensional mapping to generate a high-dimensional feature space matrix.

[0010] The high-dimensional feature space matrix is ​​input into a three-dimensional human posture recognition model that integrates motion constraints and frequency-division spatiotemporal features to obtain a three-dimensional human posture key point sequence, and three-dimensional human posture estimation is achieved through three-dimensional coordinates.

[0011] The process of obtaining the 3D human pose keypoint sequence includes: extracting spatial feature information of keypoints from a high-dimensional feature matrix using a frequency-division spatial Transformer encoder to obtain a spatial feature matrix; reshaping the spatial feature matrix to convert it into a temporal tensor; extracting temporal features using a frequency-division temporal Transformer encoder; processing the temporal features using a multilayer perceptron decoder; and outputting the final 3D human pose keypoint sequence. The 3D human pose estimation model includes a frequency-division Transformer spatiotemporal module and a multilayer perceptron decoder.

[0012] As a preferred embodiment of the three-dimensional human pose estimation method of the present invention, the nonlinear high-dimensional mapping includes using a multilayer perceptron (MLP) as the mapping function, taking a low-dimensional feature vector as input, performing a linear transformation through a fully connected layer, and outputting a dimension of D, which is achieved by adjusting the number of neurons in the last fully connected layer.

[0013] The two-dimensional human pose key point sequence is divided into a matrix sequence. ,Will Mapping to a c-dimensional high-dimensional feature space and performing coordinate encoding yields a c-dimensional high-dimensional feature space matrix.

[0014] As a preferred scheme of the three-dimensional human posture estimation method, the spatial Transformer encoder comprises a frequency division Transformer space-time module, the frequency division Transformer space-time module comprises a frequency division spatial Transformer encoder and a frequency division time Transformer encoder, the frequency division time Transformer encoder has the same structure as the frequency division spatial Transformer encoder; the spatial features are extracted by using the frequency division spatial Transformer encoder module, the spatial features are divided into four parts according to a frequency spectrum diagram by using a reinforcement learning driven adaptive unsupervised segmentation module, self-attention Transformer feature extraction is respectively performed on the four parts, feature information in different spatial frequency domains is obtained, and then the feature information is spliced to obtain the spatial features.

[0015] The frequency division time Transformer encoder extracts frequency division time features, and the frequency division time features comprise:

[0016] The high-frequency time sequence branch is used for extracting fast motion joint point features, and the low-frequency time sequence branch is used for modeling slow motion joint point features.

[0017] The time sequence features are input into a decoder of geometric kinematics constraint optimization to generate physically reliable three-dimensional human posture key points.

[0018] As a preferred scheme of the three-dimensional human posture estimation method, the three-dimensional human posture recognition model comprises a loss function, the training process is judged according to a calculation result of the loss function, and the parameters of the loss function are dynamically adjusted in the training process.

[0019] In the training process, the average error is evaluated once every training cycle on a verification set, if the average error does not decrease for consecutive cycles, the weights of the subtasks in the loss function are adjusted. The adjacent joint spacing and the angle penalty term are calculated by using a generation network of dynamic adjacent bone lengths and directions, the relative speed change of the joint points is calculated to construct a relative speed loss, and the loss function further comprises a physical-geometric collaborative constraint loss, the function is used for fusing kinematics geometric constraints and dynamics physical constraints to improve the rationality and robustness of the posture estimation.

[0020] As a preferred scheme of the three-dimensional human posture estimation method, the loss function comprises a position loss function, a bone length loss function, a bone angle loss function and a relative speed loss function, the loss function is obtained by weighted summation of the position loss function, the bone length loss function, the bone angle loss function and the relative speed loss function, the difference of the speeds of the body parts in adjacent predicted frames is considered to promote the smoothness of the 3D posture sequence.

[0021] As a preferred scheme of the three-dimensional human posture estimation method, the loss function comprises a position loss function, a bone length loss function, a bone angle loss function and a relative speed loss function, the loss function is obtained by weighted summation of the position loss function, the bone length loss function, the bone angle loss function and the relative speed loss function, the difference of the speeds of the body parts in adjacent predicted frames is considered to promote the smoothness of the 3D posture sequence.

[0022] ​Stop iterating training the monocular three-dimensional human pose estimation model when the loss function tends to be stable;The frequency division time Transformer encoder includes a frequency division attention operation layer, a batch normalization layer, a Relu layer and a multi-layer perception decoder.

[0023] As a preferred scheme of the three-dimensional human pose estimation method, the processing by the multi-layer perception decoder includes extracting spatial frequency domain feature information of the key points by the improved wavelet transform, inputting the frequency domain feature information into an adaptive unsupervised segmentation algorithm driven by reinforcement learning to obtain four different frequency division sub-band matrices, inputting the sub-band matrix features into the frequency division space Transformer encoder respectively, extracting spatial feature information of different frequency domain sub-bands respectively, and obtaining the spatial feature information after splicing.

[0024] Specifically, a multi-scale decomposition strategy is designed for the spatial frequency domain characteristics of the human pose key points by the improved wavelet transform algorithm, and a frequency division sub-band matrix with spatial-frequency joint representation capability is generated;The improved wavelet transform algorithm adopts a direction-sensitive filter bank to enhance the frequency domain feature capture capability of dynamic gestures;The adaptive unsupervised segmentation algorithm driven by reinforcement learning models the segmentation process as a Markov decision process, and the segmentation algorithm selects appropriate segmentation actions according to the frequency domain and spatial feature distribution of the data, including dividing boundaries and merging regions;A multi-objective reward function is designed to comprehensively consider the frequency domain consistency and spatial continuity of the segmentation region.

[0025] As a preferred scheme of the three-dimensional human pose estimation method, the three-dimensional human pose estimation by the three-dimensional coordinates includes reshaping the spatial feature information to a time sequence dimension tensor, and inputting the time sequence dimension tensor into a frequency division time Transformer encoder that fuses geometric kinematics constraints;Through a high-frequency time sequence branch, fast motion joint feature is extracted;Through a low-frequency time sequence branch, slow motion joint feature is extracted.

[0026] The finally extracted spatio-temporal feature information is input into a geometric dynamics optimization decoder to output a final three-dimensional human pose key point sequence, each key point is represented by a three-dimensional coordinate, and three-dimensional human pose estimation is realized by the three-dimensional coordinate.

[0027] In a second aspect, the application provides a three-dimensional human pose estimation system, which includes a two-dimensional human pose estimation module responsible for extracting two-dimensional human pose data from a video picture sequence, estimating 17 key point coordinates of each frame by a multi-scale feature fusion deep learning network, and generating a two-dimensional pose key point sequence;

[0028] The high-dimensional feature mapping module maps the two-dimensional pose key point coordinates to a high-dimensional space to generate a high-dimensional feature matrix; this step realizes the promotion and position coding of two-dimensional coordinates through coordinate coding and feature space mapping;

[0029] The three-dimensional pose estimation module adopts a frequency division Transformer space-time module, which includes a frequency division space Transformer encoder and a frequency division time Transformer encoder; an adaptive unsupervised segmentation algorithm driven by reinforcement learning is used to perform frequency domain decoupling segmentation on the high-dimensional feature matrix and extract spatial and time sequence features; after multiple processing steps, a multi-layer perceptron decoder is used to obtain the final three-dimensional human pose key point sequence;

[0030] The training optimization module optimizes the monocular three-dimensional human pose estimation model through training and validation datasets; by calculating the position loss, geometric loss and relative velocity loss function, the model parameters are adjusted to reduce the loss and improve the accuracy; the human body limb centroid velocity calculation method is used to optimize the velocity estimation when the key points are blocked, and the accuracy of motion analysis is improved.

[0031] In a third aspect, the present application provides a computer device comprising a memory and a processor, the memory storing a computer program, wherein the computer program is executed by the processor to implement any step of the three-dimensional human pose estimation method according to the first aspect of the present application.

[0032] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by the processor to implement any step of the three-dimensional human pose estimation method according to the first aspect of the present application.

[0033] The application has the beneficial effects that: the application is mainly used for estimating three-dimensional human body posture in monocular video, and fuses geometric kinematics constraint and frequency division space-time feature, greatly improves the robustness and detection accuracy of monocular three-dimensional human body posture estimation method. Therefore, it can be quickly applied to pedestrian motion posture trend detection and other fields, such as blind area pedestrian collision warning function in engineering vehicles. A frequency division space-time Transformer encoder module fusing geometric kinematics constraint is proposed, the key point space features of different motion frequencies are extracted through the frequency division space Transformer encoder, then the spliced features are sent into the frequency division time Transformer encoder to extract the information on time sequence, further improve the detection accuracy, and to a certain extent, improve the anti-jitter on time sequence. The application proposes a new network training loss function, which not only focuses on the error value between the final three-dimensional key points and the real key points, but also calculates the error value for the relative motion speed, bone length and bone direction of the key points, so the training is easier to converge and the training process is more stable. In the prior art, the three-dimensional posture of the key frame, i.e. the intermediate frame, is obtained through a single layer perception machine, while the method can obtain multiple frame results at the same time by setting the Transformer encoder and the fully connected convolutional layer. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0035] Figure 1 Flowchart of three-dimensional human body posture estimation method.

[0036] Figure 2 Posture recognition flowchart of three-dimensional human body posture estimation method.

[0037] Figure 3 Human body key point diagram of three-dimensional human body posture estimation method.

[0038] Figure 4 Training flowchart of three-dimensional human body posture estimation method. DETAILED DESCRIPTION

[0039] In order to make the above-mentioned purposes, features and advantages of the application more obvious and easy to understand, the specific embodiments of the application will be described in detail below with reference to the drawings of the specification.

[0040] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present application.

[0041] Secondly, the "one embodiment" or "an embodiment" referred to herein means a specific feature, structure, characteristic, or combination of features and / or characteristics described herein that can be included in at least one implementation of the present application. The appearance of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all referring to a single, special implementation of the present application, and the different instances can not be mutually exclusive.

[0042] Reference is made to Figures 1-4 For one embodiment of the present application, the embodiment provides a three-dimensional human pose estimation method, comprising the following steps:

[0043] S1: Extracting two-dimensional human pose key points from a monocular video picture sequence to generate a two-dimensional human pose key point sequence.

[0044] Further, the two-dimensional human pose estimation is performed on the dataset in the video picture sequence to obtain key point information and map it to a high-dimensional space to obtain a high-dimensional feature space matrix.

[0045] Specifically, the two-dimensional human pose estimation on the dataset in the video picture sequence in S1 includes the following steps:

[0046] Step 1.1, preprocessing the three-dimensional human pose estimation dataset to obtain a picture sequence including two-dimensional human poses and corresponding three-dimensional human pose information.

[0047] The preprocessing process includes data cleaning, noise removal, image correction, or other necessary data preparation work, and the goal is to ensure the quality and consistency of the dataset.

[0048] Step 1.2, inputting the above obtained two-dimensional human pose picture sequence into a two-dimensional human pose detector to obtain a two-dimensional human pose key point skeleton, including a sequence of 17 two-dimensional human pose key points.

[0049] Two-dimensional human pose detection usually adopts an end-to-end model architecture based on deep learning. In the embodiment, a multi-scale feature fusion network with high precision and real-time balance advantage is selected as the backbone model. The network is based on an improved HRNet (High-Resolution Network) architecture, and the spatial detail information is maintained through parallel multi-resolution subnetwork interaction. HRNet usually adopts a ResNet-50 backbone network with a resolution of 384*288. The main task is to accurately identify and locate the human key points from the image.

[0050] As shown in Figure 3 , the detection process outputs a standardized human joint skeleton topology, including 17 anatomical landmark points of the head (nose tip, eyes, ears), torso (shoulders, elbows, wrists, spine midpoint), and lower limbs (hips, knees, ankles). This process achieves sub-pixel level positioning accuracy through the joint optimization of Heatmap Regression and coordinate offset prediction, and finally generates a key point sequence: , where f is the frame number of the corresponding input image, for example, the commonly used frame numbers f are 300, 243, 128, 81, etc. J is the 17 key point coordinates. The subscript 2 is the corresponding two-dimensional coordinate value .

[0051] Step 1.3, divide the sequence of two-dimensional human pose key points obtained above. The commonly used division step is 243, 128, 81, etc. The present application uses a step size of 243 to divide the above sequence.

[0052] The two-dimensional spatial point coordinates are lifted to a c-dimensional feature space and are encoded to obtain a c-dimensional feature matrix , that is:

[0053]

[0054] , where is a conversion matrix that converts the dimension from 2 to c, is a coordinate encoding matrix, which is a learnable matrix that can encode features and increase the position information of the features.

[0055] It should be noted that in the above S1, lifting the obtained two-dimensional human pose to a high-dimensional space specifically includes dividing the obtained matrix sequence , where J is generally 17 and f is generally 243. Map to a c-dimensional feature space. Generally, the c-dimensional is a high-dimensional, which is mainly used to extract more feature information in the subsequent bidirectional space-time module. Common c-dimensional are 128, 256, 512, etc.

[0056] S2: Project the two-dimensional human pose key point sequence into a feature space through a nonlinear high-dimensional mapping to generate a high-dimensional feature space matrix.

[0057] Further, as shown in Figure 2 , the high-dimensional feature space matrix is input into a monocular three-dimensional human pose estimation model for processing. Specifically, the monocular three-dimensional human pose estimation model includes a frequency division Transformer space-time module and a multi-layer perception decoder, and the processing process is as follows:

[0058] First, the spatial feature information of the key points is extracted from the high-dimensional feature space matrix through the frequency division space Transformer encoder to obtain a spatial feature matrix; the spatial feature matrix is reshaped to a time sequence dimension tensor, and then the time sequence features are extracted through the frequency division time Transformer encoder, and the final three-dimensional human pose key point sequence is output through the multi-layer perception decoder to realize three-dimensional human pose estimation.

[0059] Specifically, the frequency division Transformer space-time module in S2 above includes a frequency division space Transformer encoder and a frequency division time Transformer encoder, wherein the frequency division time Transformer encoder has the same structure as the frequency division space Transformer encoder, except that the input data is different.

[0060] In the frequency division space Transformer encoder, there is a reinforcement learning driven adaptive unsupervised segmentation algorithm module, a spatial attention operation layer, a batch normalization layer, a Relu layer, and a multi-layer perception decoder. In the spatial attention operation layer, a multi-head attention mechanism is used to calculate the attention of each head and fuse the features of each head to obtain the final multi-head attention. The batch normalization (BN) method is an effective layer-by-layer normalization method that can normalize the intermediate layer. The main advantage is that normalization makes the optimization data smoother.

[0061] In particular, in the frequency division space Transformer encoder, the first step needs to implement frequency domain decoupling segmentation on the input spatial feature tensor through a reinforcement learning driven adaptive unsupervised segmentation algorithm module. Specifically, by improving the wavelet transform , the input high-dimensional feature data is mapped to the frequency domain space, and then a reinforcement learning driven adaptive unsupervised segmentation algorithm module is used to generate four sub-band coefficient matrices . The low-frequency component is concentrated in the upper left corner of the frequency spectrum, encodes the overall contour and joint topology of the human body, and corresponds to the overall structure information of the image; the horizontal high-frequency component corresponding to vertical edges, reflecting the limb's longitudinal movement tendency; vertical high-frequency components corresponding to horizontal edges, representing the limb's lateral movement characteristics; diagonal high frequencies concentrated in the diagonal position of the frequency spectrum, encoding details and local deformations.

[0062] wherein the improved wavelet transform algorithm a multi-scale decomposition strategy is designed for the spatial frequency domain characteristics of human posture key points to generate a frequency sub-band matrix with spatial-frequency domain joint representation capability; wherein the improved wavelet transform uses a direction-sensitive filter bank to enhance the frequency domain feature capture capability of dynamic posture, and by designing multiple direction filters (such as 0°, 45°, 90°, 135°), the feature information of a specific direction is extracted. The formula is expressed as:

[0063]

[0064] wherein is the direction basis function, generally using Gabor wavelet, D is the action direction in human posture estimation. represents the wavelet coefficient of the signal after wavelet transform at the scale, the position.

[0065] Here, a reinforcement learning driven adaptive unsupervised segmentation algorithm module is needed to divide the four sub-band coefficient matrices. The commonly used method is to directly divide the frequency spectrum into four equal size modules, with a length and width of 1 / 2 of the frequency spectrum. This segmentation method has certain defects.

[0066] From the perspective of data characteristics, the information distribution in the frequency spectrum is often uneven, and different regions may carry information of different importance and characteristics. Equal size segmentation cannot fully consider the differences in this information distribution, which may lead to important information being improperly segmented into different modules, or regions with similar characteristics being artificially split apart, thereby destroying the internal structure and relevance of the data. Therefore, a reinforcement learning driven adaptive unsupervised segmentation algorithm is proposed. The main steps are:

[0067] This algorithm models the segmentation process as a Markov decision process, and the segmentation algorithm interacts with the environment (different frequency domain sub-band data) to select appropriate segmentation actions (such as dividing boundaries, merging regions, etc.) based on the current state (data frequency domain and spatial feature distribution). After each action, the system will feedback a reward signal, which is evaluated according to the accuracy of the segmentation result, regional consistency, etc.

[0068] The segmentation algorithm adjusts its strategy according to the reward signal through trial and error and learning, so as to maximize the long-term cumulative reward, and thus adaptively finds the optimal segmentation mode to extract the spatial feature information of different frequency domain subbands. The multi-objective reward function comprehensively considers the frequency domain consistency, spatial continuity and semantic integrity of the segmentation region.

[0069] The multi-objective reward function mainly evaluates the reward from two dimensions of frequency domain consistency and spatial consistency. Specifically:

[0070] The Fourier transform of the segmentation region is calculated to extract the amplitude spectrum. The frequency spectrum entropy is calculated:

[0071]

[0072] wherein is the normalized frequency spectrum energy.

[0073] The frequency domain reward function is:

[0074] The boundary gradient of the segmentation region is calculated: the Sobel operator is used to extract the edge. The edge continuity is calculated. The spatial reward function is:

[0075]

[0076] Multi-objective reward function (comprehensive):

[0077]

[0078] wherein , are weight coefficients, initially set to 0.5, 0.5. In the model training stage, adaptive learning can be achieved through gradient descent. represents the length of the continuous edge, represents the total edge length.

[0079] The adaptive unsupervised segmentation algorithm module adjusted by the multi-objective reward function obtains , the formula is:

[0080]

[0081] In the above formula, represents the adaptive unsupervised segmentation algorithm module driven by reinforcement learning. The algorithm module contains a multi-objective reward function for segmentation effect evaluation. represents the improved two-dimensional wavelet transform function. Next, a multi-head attention mechanism and a multi-layer perceptron decoder are used to process the four subband coefficient matrix features, and the specific processing process is as follows:

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090] where LN is Layer Normalization, MSA is Multi-Head Self-Attention, and MLP is Multi-Layer Perceptron decoder. represents the result obtained by the first linear layer of the Multi-Layer Perceptron decoder, represents the result obtained by the second linear layer of the l-th layer of the Multi-Layer Perceptron.

[0091] After processing the characteristics of the four sub-band matrices, the , , , are spliced to obtain the final result:

[0092] , ,

[0093] In the above formula, is the final feature matrix obtained after splicing the sub-band matrix features, and the number of channels is the sum of the channel numbers of all sub-band features. Concate() is a splicing function that stacks the input N sub-band feature matrices in order along the channel dimension to generate a fused multi-scale feature representation.

[0094] In the temporal Transformer module, the feature layer is divided into two modules, i.e., high-speed and low-speed modules, by motion speed, and the speed threshold for segmentation is which is a trainable threshold parameter. The subsequent operations are the same as the spatial Transformer, which will not be described in detail here.

[0095] In the multi-head attention mechanism, the correlation features of each head are obtained by extracting the correlation features of Q, K, and V through the Attention multi-head attention mechanism. Generally, 8 heads are more commonly used, among which . ​

[0096]

[0097]

[0098] wherein, The function is to average the input features Y into 3 parts to obtain Q, K, and V. In addition, the attention formula is represented as:

[0099]

[0100] The features obtained by each head attention mechanism are combined, and finally the feature information output by the attention mechanism is obtained.

[0101] In an embodiment, the S2 specifically comprises the following sub-steps:

[0102] Step 2.1, the features obtained by the spatial Transformer encoder are reshaped into a spatial dimension tensor and then reshaped into a time sequence dimension tensor and input into the frequency division time Transformer encoder to extract time sequence features.

[0103] Step 2.2, in the frequency division time sequence features, the high-speed motion key point time sequence features and the low-speed key point time sequence features are divided, which correspond to the key point trajectory time sequence features of different motion speeds. When performing forward time sequence feature extraction, the time sequence dimension tensor is input into the time Transformer encoder, and the structure of the time Transformer encoder is the same as that of the spatial Transformer encoder, except that the input data is different.

[0104] It should be noted that in the multi-head attention mechanism calculation in the Transformer, the attention of each head is calculated first, the features of each head are fused to obtain the final multi-head attention, and then the features are sent to the regularization layer for data regularization. Then, through the multi-layer perception decoder, the obtained two parts of the space-time self-attention state are updated separately to obtain the space-time feature matrix, and are regularized.

[0105] S3: input the high-dimensional feature space matrix into a monocular three-dimensional human pose estimation model with geometric kinematic constraints to obtain a three-dimensional human pose key point sequence, and realize three-dimensional human pose estimation through three-dimensional coordinates.

[0106] Further, the process of extracting spatial correlation features through the Transformer bidirectional space-time module mainly includes reshaping the high-dimensional feature space matrix obtained by S1 to project it into a spatial dimension tensor ​That is, from 243*17*512 to 17*243*512, and then the Split operation is performed on the mapped matrix, that is, Q, K, V = Split(Y), so the feature dimensions of Q, K, V are 17*81*512.

[0107] The spatial Transformer encoder is fed into the space, and the global features of the key points are extracted. Figure 2 As shown in the Transformer encoder, a spatial attention operation layer, a batch normalization layer and a Relu layer are included; in the spatial attention operation layer, a multi-head attention mechanism is adopted, the attention of each head is calculated first, and the features of each head are fused to obtain the final multi-head attention; the batch normalization (BN) method is an effective layer-by-layer normalization method, which can normalize the intermediate layer, and the main advantage is that normalization can make the optimization data smoother.

[0108] The process of extracting bidirectional time sequence correlation features through the Transformer mainly includes: reshaping the spatial information, that is, from 17*243*512 to 243*17*512, and the subsequent process is the same as the spatial Transformer, that is, a feature matrix containing time sequence information is obtained .

[0109] A multi-layer perceptron decoder is adopted, the time sequence feature information is input into a fully connected convolutional layer, the c-dimensional features in the feature space are reduced to 3-dimensional, and the final three-dimensional human pose key point sequence is obtained.

[0110] Specifically, the information on the time sequence is input into a fully connected convolutional layer to obtain the final three-dimensional human pose key point sequence, including the following specific steps:

[0111] A multi-layer perceptron decoder is adopted, the c-dimensional features in the feature space are reduced to 3-dimensional:

[0112]

[0113] The final output three-dimensional human pose sequence result is:

[0114]

[0115] This is different from most current methods and is more convenient. The present application can obtain multiple frame results at the same time, for example, when f=243, 243 frames of three-dimensional pose output results can be obtained.

[0116] The matrix feature output by the space-time module is 243*17*512, and the full connection convolution layer in this embodiment is used to reduce the dimension 512 to 3, so that the matrix output is 243*17*3, which is the final output result.

[0117] Before applying the monocular three-dimensional human pose estimation model to pose estimation, the model is trained and verified using the human pose estimation dataset.

[0118] The step flow of training is as shown in Figure 4

[0119] (1) Obtain a two-dimensional pose sequence.

[0120] (2) Perform data normalization operation.

[0121] (3) Initialize the monocular three-dimensional human pose estimation model of the frequency division space-time feature.

[0122] (4) Train the monocular three-dimensional human pose estimation model using the dataset, and iterate the monocular three-dimensional human pose estimation model.

[0123] (5) Calculate the loss function between the output result of the model and the true value.

[0124] (6) Determine whether the loss is stable.

[0125] (7) When the loss is still decreasing and not stable, repeat step (4).

[0126] (8) When the loss is stable, the training is completed, and the trained monocular three-dimensional human pose estimation model is obtained.

[0127] The position loss function, the skeletal geometry constraint function and the relative velocity loss function are set to predetermined weights, and the loss function is expressed by the following formula:

[0128]

[0129] wherein, and are weight coefficients, and the sum is 1; is the position loss function; is the geometry loss function; is the physics loss function; is the relative velocity loss function, which considers the difference in the speed of the body part in the adjacent predicted frame, and promotes the smoothness of the 3D pose sequence. When the loss function is stable, stop iterating the training of the monocular three-dimensional human pose estimation model.

[0130] Wherein, it needs to be introduced ​Relative speed loss function. The setting of the speed loss function will fail when the joint is blocked, for example, the joint of the athlete's foot is blocked during running, so the speed loss function will be empty and cannot be accurately calculated, which seriously affects the accuracy of sports analysis.

[0131] For this defect, the application provides a human body limb joint centroid speed calculation method. According to the correlation of human body limb joints, 17 joints in the human body can be divided into 5 parts: left hand area, right hand area, left leg area, right leg area and middle trunk area. The 17 joints of the human body are divided into 5 regions (left hand, right hand, left leg, right leg and trunk), and the centroid coordinates of each region are The centroid coordinates of each region are

[0132]

[0133] Wherein, respectively represent the two-dimensional coordinates of the key points and axis coordinates.

[0134] At the same time, by introducing the bird flock cooperative flight characteristics, a human body motion speed calculation model based on group behavior simulation is constructed. The scheme uses the obstacle avoidance-following mechanism in bird flock movement to optimize the speed estimation algorithm when the human body joint is blocked, and the specific improvements are as follows: combine the core dynamics equation of bird flock movement (such as the separation-alignment-aggregation model) with the kinematic constraint conditions of human body, wherein the separation behavior corresponds to the minimum safety distance constraint between joints, the alignment behavior corresponds to the consistency of adjacent frame motion trend, and the aggregation behavior corresponds to the maintenance of limb segment space connectivity, and then the improved speed calculation formula suitable for blocked scene is derived.

[0135] On the basis of the traditional Cucker-Smale group motion model, the kinematic constraint of human body is fused, and the speed update formula under the blocked scene is derived:

[0136]

[0137] Wherein, is a dynamic weight based on joint distance, is the visible neighborhood set of the i-th joint is the kinetic potential function of the limb segment; , , is the behavior weight adjustment parameter (obtained by training the CMU motion database).

[0138] The traditional loss function ​The relative speed loss function focuses on the difference between the predicted three-dimensional key point coordinates and the real three-dimensional key point coordinates The relative speed loss function focuses on the difference between the predicted three-dimensional key point coordinates and the real three-dimensional key point coordinates

[0139] It should be noted that the present application improves the detection accuracy of three-dimensional human posture by extracting frequency division space-time feature information, and further reduces the jitter of three-dimensional human posture between adjacent frames by fusing time and space features and using the method of obtaining attention between adjacent frames, and further incorporates relative speed information in the loss function, so that the training process is more robust and stable.

[0140] Embodiment 2, refer to Figure 3 An embodiment of the present application provides a multi-species collaborative construction deployment optimization method, in order to verify the beneficial effects of the present application, scientific demonstration is carried out through economic benefit calculation and simulation experiment.

[0141] The method proposed by the present application mainly completes training and testing on double Nvidia RTX2080Ti GPU display cards, the running environment is Pytorch1.8.1, Python3.8, and the data set is Human3.6M data set collected by a visual sensor. Human3.6M is one of the largest motion capture data sets, including 3.6 million 3D human postures and corresponding images. The video data contained in the data set is shot by 4 high-resolution progressive scan cameras at a speed of 50 Hz. The data set involves the activities of 11 professional actors in 17 scenes: discussion, smoking, taking pictures, making phone calls, etc., and provides accurate 3D joint positions and high-resolution videos.

[0142] As shown in Figure 3 The two-dimensional key points are the 17 key nodes of the two-dimensional human body extracted by the CPN network and the real GroundTruth. In the experimental specific parameters, N1 and N2 are the number of time and space submodules of the single-channel Transformer, which are both set to 4. The continuous frame number is set to 243, which is the commonly used frame number in related research. The batch_size is set to 1024 during training, the initial learning rate is set to 0.0004, the training times epoch is 120, and the learning rate is attenuated to 99% every training epoch.

[0143] 1) The method of dividing the Human3.6M data set into S1, S5, S7, S8 as the training set and S9, S11 as the test set.

[0144] 2) Input the video image into the trained two-dimensional human pose detector CPN to obtain a two-dimensional human skeleton key point sequence, and save it.

[0145] 3) Divide the input skeleton key point sequence into a matrix with a batch size of 1024, f=243, J=17, and a coordinate of 2.

[0146] 4) The 243*17*2 matrix obtained in 3) is input into a neural network linear layer with an input dimension of 2 and an output dimension of 512 to obtain a 243*17*512 matrix, and then a position encoding matrix 17*512 is added to each frame feature.

[0147] 5) The 243*17*512 obtained in 4) is reshaped to map to a high spatial dimension, i.e., 17*243*512.

[0148] 6) The matrix obtained in 5) is input into a depth-8 Transformer encoding layer. Each Transformer encoding layer first divides the input matrix into the same 3 matrix features, i.e., through the Split function to obtain Q, K, and V, with dimensions of 17*81*512. Then calculate the multi-head attention mechanism, in this embodiment, 8 heads are used, i.e., through 8 different self-attention mechanisms to capture features of different scales, and each attention mechanism is scaled to prevent dimension from being too large to cause the activation layer to produce bias. Then fuse these attention mechanisms to obtain a 17x512 feature matrix, and finally pass the result through a regularization layer with a feature dimension of 512, and then input it into a 2-layer perceptron layer. The first layer raises the 512-dimensional feature to a 4x512 layer, then passes it through a sigmoid activation layer and a dropout layer, and then reduces the 4x512 feature to a 512 layer, and then passes it through a sigmoid activation layer and a dropout layer. In this embodiment, the features before entering the Transformer and the features through the multi-head attention and the features through the multi-layer perceptron are connected in residual to prevent gradient vanishing and gradient explosion.

[0149] 7) Input the bidirectional spatio-temporal information obtained in step 6, with a matrix dimension of 243*17*512, into a fully connected layer with an input of 512 and an output of 3, to obtain an output matrix with a matrix dimension of 243*17*3.

[0150] To verify the effectiveness and practicability of the method, a verification example on the Human3.6M dataset is given below, and Table 1 is the detection result of the example on the test set, and the measurement standard is MPJPE (Mean Per Joint Position Error) which is the average error value of all key points. The full name and abbreviation of the 15 action categories in the dataset are defined as: Directions (Dir), Discussion (Disc), Eating (Eat), Greeting (Greet), Talking on the phone (Phone), Taking photo (Photo), Posing (Pose), Making purchases (Purch), Sitting (Sit), Sitting Down (SitD), Smoking (Smoke), Waiting (Wait), Walking dog (WalkD), Walking (Walk), and Walking together (WalkT).

[0151] Table 1 Comparison table of action effect of the application

[0152]

[0153] The application selects the current mainstream method in recent years for comparison, as shown in Table 1, which is the result of the application method on the Human3.6M dataset using MPJPE error for evaluation.

[0154] The two-dimensional joint points compared are detected by the CPN network, the comparison result is the MPJPE error, the comparison dataset is Human3.6M, and the bold part is the best estimation effect. According to the results in Table 1, 15 common actions are compared, and it can be found that the MPJPE error of the method provided by the application is 41.6mm, which is better than other methods.

[0155] The embodiment also provides a three-dimensional human pose estimation system, comprising:

[0156] A two-dimensional human pose estimation module is responsible for extracting two-dimensional human pose data from a video picture sequence, estimating 17 key point coordinates of each frame through a multi-scale feature fusion deep learning network, and generating a two-dimensional pose key point sequence.

[0157] A high-dimensional feature mapping module maps the two-dimensional pose key point coordinates to a high-dimensional space to generate a high-dimensional feature space matrix; this step realizes the promotion and position coding of the two-dimensional coordinates through coordinate coding and feature space mapping.

[0158] The three-dimensional pose estimation module adopts a frequency division Transformer space-time module, and the module includes a frequency division space Transformer encoder and a frequency division time Transformer encoder; an adaptive unsupervised segmentation algorithm driven by reinforcement learning is used to perform frequency domain decoupling segmentation on a high-dimensional feature matrix and extract spatial and time sequence features; after multiple processing steps, a multi-layer perceptron decoder is used to obtain a final three-dimensional human pose key point sequence.

[0159] The training optimization module optimizes the monocular three-dimensional human pose estimation model through training and validation datasets; model parameters are adjusted to reduce loss and improve accuracy by calculating position loss, geometric loss, and relative velocity loss functions; the human limb centroid velocity calculation method is used to optimize velocity estimation when the joint is blocked, improving the accuracy of motion analysis.

[0160] The embodiment also provides a computer device suitable for the three-dimensional human pose estimation method, including a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to realize the three-dimensional human pose estimation method proposed in the above embodiment.

[0161] The computer device can be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through WIFI, an operator network, NFC (near field communication), or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad arranged on the shell of the computer device. In addition, the input device can be an external keyboard, touchpad, or mouse, etc.

[0162] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the three-dimensional human pose estimation method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0163] In summary, this invention achieves the following: Extracting two-dimensional human pose key points from a monocular video image sequence to generate a two-dimensional human pose key point sequence; projecting the two-dimensional human pose key point sequence onto a feature space through a nonlinear high-dimensional mapping to generate a high-dimensional feature space matrix; inputting the high-dimensional feature space matrix into a geometrically kinematically constrained monocular three-dimensional human pose estimation model to obtain a three-dimensional human pose key point sequence, and realizing three-dimensional human pose estimation through three-dimensional coordinates; obtaining the three-dimensional human pose key point sequence includes: extracting frequency domain features of the image through an improved wavelet transform, and then generating four sub-band coefficient matrices through a reinforcement learning-driven adaptive unsupervised segmentation algorithm module; inputting the frequency band sub-matrices into a spatial Transformer encoder to extract and combine the spatial features in the frequency domain; reshaping the extracted spatial feature information into a temporal dimension tensor; extracting joint features of fast and slow motions using high-frequency and low-frequency temporal branches respectively; inputting the extracted spatiotemporal features into a decoder, and outputting the three-dimensional human pose key point sequence after optimization.

[0164] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A three-dimensional human pose estimation method, characterized in that: This includes extracting two-dimensional human pose key points from monocular video image sequences and generating two-dimensional human pose key point sequences. The two-dimensional human posture key point sequence is projected onto the feature space through a nonlinear high-dimensional mapping to generate a high-dimensional feature space matrix. The high-dimensional feature space matrix is ​​input into a three-dimensional human posture recognition model that integrates motion constraints and frequency-division spatiotemporal features to obtain a three-dimensional human posture key point sequence, and three-dimensional human posture estimation is achieved through three-dimensional coordinates. The process of obtaining the 3D human pose keypoint sequence includes: extracting spatial feature information of keypoints from a high-dimensional feature matrix using a frequency-division spatial Transformer encoder to obtain a spatial feature matrix; reshaping the spatial feature matrix to convert it into a temporal tensor; extracting temporal features using a frequency-division temporal Transformer encoder; processing the temporal features using a multilayer perceptron decoder; and outputting the final 3D human pose keypoint sequence. The 3D human pose estimation model includes a frequency-division Transformer spatiotemporal module and a multilayer perceptron decoder. The processing via the multilayer perceptron decoder includes: extracting the spatial frequency domain feature information of key points through an improved wavelet transform; inputting the frequency domain feature information into an adaptive unsupervised segmentation algorithm driven by reinforcement learning to obtain four different frequency sub-band matrices; inputting the sub-band matrix features into the frequency-division spatial Transformer encoder to extract the spatial feature information of different frequency sub-bands; and then concatenating the sub-bands to obtain the spatial feature information. By employing an improved wavelet transform algorithm, a multi-scale decomposition strategy is designed to address the spatial-frequency domain characteristics of key human posture points, generating a frequency-divided sub-band matrix with joint spatial-frequency domain representation capabilities. The improved wavelet transform algorithm utilizes a direction-sensitive filter bank to enhance the capture of frequency domain features of dynamic postures. A reinforcement learning-driven adaptive unsupervised segmentation algorithm models the segmentation process as a Markov decision process, selecting appropriate segmentation actions based on the frequency domain and spatial feature distribution of the data, including boundary delineation and region merging. A multi-objective reward function is designed to comprehensively consider the frequency domain consistency and spatial continuity of the segmented regions. The method of achieving 3D human pose estimation through 3D coordinates includes: reshaping the spatial feature information into a temporal dimensional tensor, and inputting it into a frequency-division time Transformer encoder that integrates geometric kinematic constraints; extracting fast motion joint features through high-frequency temporal branches; and extracting slow motion joint features through low-frequency temporal branches. The extracted spatiotemporal feature information is input into the geometric dynamics optimization decoder, and the final 3D human pose key point sequence is output. The 3D coordinates of each key point are represented by the 3D human pose key point sequence, and 3D human pose estimation is achieved through the 3D coordinates.

2. The three-dimensional human pose estimation method as described in claim 1, characterized in that: The nonlinear high-dimensional mapping includes using a multilayer perceptron (MLP) as the mapping function, taking a low-dimensional feature vector as input, performing a linear transformation through a fully connected layer, and outputting a dimension of D, which is achieved by adjusting the number of neurons in the last fully connected layer. The two-dimensional human posture key point sequence is divided into a matrix sequence X. X is then mapped to a c-dimensional high-dimensional feature space and encoded into coordinates to obtain a c-dimensional high-dimensional feature space matrix.

3. The three-dimensional human pose estimation method as described in claim 2, characterized in that: The spatial Transformer encoder includes a frequency-division Transformer spatiotemporal module comprising a frequency-division spatial Transformer encoder and a frequency-division time Transformer encoder, wherein the frequency-division time Transformer encoder has the same structure as the frequency-division spatial Transformer encoder; the frequency-division spatial Transformer encoder module is used to extract spatial features, and a reinforcement learning-driven adaptive unsupervised segmentation module is applied to divide the spatial features into four parts according to the spectrogram, and self-attention Transformer feature extraction is performed on each part to obtain feature information in different spatial frequency domains, which are then concatenated to obtain the spatial features; The frequency division time Transformer encoder extracts frequency division time features, including: The high-frequency temporal branch is used to extract features of fast-moving joints; the low-frequency temporal branch is used to model features of slow-moving joints. Temporal features are input into a decoder optimized by geometric kinematic constraints to generate physically reliable 3D human pose key points.

4. The three-dimensional human pose estimation method as described in claim 3, characterized in that: The three-dimensional human pose recognition model includes constructing a loss function, judging the training progress based on the calculation result of the loss function, and dynamically adjusting the parameters of the loss function during the training process. During training, the average error is evaluated on the validation set every n training epochs. If the error does not decrease for n / 2 consecutive epochs, the weights of the sub-tasks in the loss function are adjusted. The distance between adjacent joints and the angle penalty term are calculated by generating a network that dynamically considers the length and direction of adjacent bones; the relative velocity loss is constructed by calculating the relative velocity change of joints. The constructed loss function also includes: physical-geometric co-constraint loss, which improves the rationality and robustness of attitude estimation by integrating kinematic geometric constraints and dynamic physical constraints.

5. The three-dimensional human pose estimation method as described in claim 4, characterized in that: The loss function is obtained by weighted summation of position loss function, bone length loss function, bone angle loss function, and relative velocity loss function; it considers the difference in velocity of body parts in adjacent prediction frames to promote the smoothness of 3D pose sequences; When the loss function tends to stabilize, the iterative training of the monocular 3D human pose estimation model is stopped; the frequency division time Transformer encoder includes a frequency division attention operation layer, a batch normalization layer, a ReLU layer and a multilayer perceptron decoder.

6. A three-dimensional human pose estimation system, based on the three-dimensional human pose estimation method according to any one of claims 1 to 5, characterized in that: It includes a 2D human pose estimation module, which is responsible for extracting 2D human pose data from video image sequences, estimating the coordinates of 17 key points in each frame through a deep learning network that fuses multi-scale features, and generating a 2D pose key point sequence. The high-dimensional feature mapping module maps the coordinates of two-dimensional pose key points to a high-dimensional space, generating a high-dimensional feature matrix; through coordinate encoding and feature space mapping, it realizes the enhancement and position encoding of two-dimensional coordinates; The 3D pose estimation module adopts a frequency-division Transformer spatiotemporal module, which includes a frequency-division spatial Transformer encoder and a frequency-division temporal Transformer encoder. A reinforcement learning-driven adaptive unsupervised segmentation algorithm is used to perform frequency domain decoupling segmentation on a high-dimensional feature matrix and extract spatial and temporal features. After multiple processing steps, the final 3D human pose keypoint sequence is obtained using a multilayer perceptron decoder. The training optimization module optimizes the monocular 3D human pose estimation model by training and validating datasets; it adjusts model parameters to reduce losses and improve accuracy by calculating position loss, geometric loss, and relative velocity loss functions; it optimizes velocity estimation when joint occlusion by using a human limb center of mass velocity calculation method; and it improves the accuracy of motion analysis by simulating bird flock cooperative behavior to optimize velocity estimation when joint occlusion.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the three-dimensional human pose estimation method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the three-dimensional human pose estimation method according to any one of claims 1 to 5.

Images