Pose Estimation Method, Device, Terminal and Medium Based on Large-Core Decoupled Convolution

Abstract

The present invention discloses a pose estimation method, device, terminal and medium based on large kernel decoupled convolution, which relates to the field of computer vision. The method maps a two-dimensional human pose sequence to a high-dimensional feature space to determine an initial three-dimensional human pose sequence; based on depthwise separable decoupled convolution, the initial three-dimensional human pose sequence is learned to determine a target three-dimensional human pose sequence, and the depthwise separable decoupled convolution is obtained by deconstructing and reconstructing the depthwise separable convolution; a multi-layer perceptron is used to determine the three-dimensional human pose based on the target three-dimensional human pose sequence. The present invention uses depthwise separable decoupled convolution to independently learn the feature relationships in the three dimensions of time, space and features, effectively solving the problem that the existing technology is difficult to be popularized and used in actual scenarios due to its high computational requirements and inability to estimate the human pose of the entire video sequence.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and particularly to a pose estimation method, apparatus, terminal, and medium based on large kernel decoupled convolution. Background Art

[0002] 3D human pose estimation aims to analyze the state and behavior of humans in complex real-world environments and can accurately predict the coordinates of human key points in 3D space. Compared with 2D pose estimation, 3D human pose estimation provides more comprehensive and accurate 3D space information, thus helping to better understand complex human behaviors.

[0003] Existing 3D human pose estimation methods based on the attention mechanism usually regard each video frame as a pose representation and use extremely long video sequences to achieve superior performance. However, since the complexity of self-attention grows quadratically with respect to the number of tokens, these methods are inevitably affected by high computational requirements, hindering the deployment of these methods on devices with limited computational resources; while 3D human pose estimation methods based on traditional convolution can only estimate the 3D human pose of the central frame of the video sequence and cannot estimate the human pose of the entire video sequence, making it equally difficult to apply to actual scenarios.

[0004] In summary, existing 3D human pose estimation methods are difficult to be popularized and used in actual scenarios due to their high computational requirements and inability to estimate the human pose of the entire video sequence.

[0005] Therefore, there is still room for improvement and development in the prior art. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a pose estimation method, apparatus, terminal, and medium based on large kernel decoupled convolution for the above-mentioned defects of the prior art, aiming to solve the problem that existing technologies are difficult to be popularized and used in actual scenarios due to their high computational requirements and inability to estimate the human pose of the entire video sequence.

[0007] The technical solution adopted by the present invention to solve the problem is as follows:

[0008] In a first aspect, an embodiment of the present invention provides a pose estimation method based on large kernel decoupled convolution, wherein the method includes:

[0009] Obtain a 2D human pose sequence of a target user, map the 2D human pose sequence to a high-dimensional feature space, and determine an initial 3D human pose sequence F0 ∈ R B×T×J×C , where R is the set of real numbers, B is the batch size, T is the time length, J is the number of joint points, and C is the feature dimension;

[0010] Learning is performed on the initial three-dimensional human pose sequence based on depthwise separable decoupled convolution to determine a target three-dimensional human pose sequence, where the depthwise separable decoupled convolution is obtained by deconstructing and reconstructing a depthwise separable convolution;

[0011] A multi-layer perceptron is used to learn the target three-dimensional human pose sequence to determine the three-dimensional human pose of the target user.

[0012] In one implementation method, the depthwise separable decoupled convolution includes a temporal large kernel convolution, a spatial large kernel convolution, and a feature large kernel convolution. Learning is performed on the initial three-dimensional human pose sequence based on the depthwise separable decoupled convolution to determine a target three-dimensional human pose sequence, including:

[0013] Learning is performed on the initial three-dimensional human pose sequence based on the temporal large kernel convolution to determine a first human pose sequence;

[0014] Learning is performed on the first human pose sequence based on the spatial large kernel convolution to determine a second human pose sequence;

[0015] Learning is performed on the second human pose sequence based on the feature large kernel convolution to determine the target three-dimensional human pose sequence.

[0016] In one implementation method, learning is performed on the initial three-dimensional human pose sequence based on the temporal large kernel convolution to determine a first human pose sequence, including:

[0017] The initial three-dimensional human pose sequence is subjected to dimension recombination and feature merging to determine a first fusion feature F1 ∈ R B×(J×C)×T ;

[0018] Reflection padding is performed on the first fusion feature based on the convolution kernel size of the temporal large kernel convolution;

[0019] The temporal large kernel convolution is used to learn the reflected padded first fusion feature to determine the first human pose sequence.

[0020] In one implementation method, the temporal large kernel convolution is implemented based on a one-dimensional convolution with a convolution kernel size of 17×17 and a number of groups equal to the product of the number of joint points and the feature dimension.

[0021] In one implementation method, learning is performed on the first human pose sequence based on the spatial large kernel convolution to determine a second human pose sequence, including:

[0022] The first human pose sequence is dimensionally reduced so that the first human pose sequence has the same dimension as the initial three-dimensional human pose sequence;

[0023] Normalize, perform dimensional recombination, and feature merging on the first human body pose sequence after dimensional reduction to determine the second fused feature F2 ∈ R B×(J×C)×T ;

[0024] Use the spatial large kernel convolution to learn the second fused feature to determine the second human body pose sequence, where the spatial large kernel convolution is implemented based on a one - dimensional convolution with the number of groups equal to the number of joint points.

[0025] In an implementation method, learn the second human body pose sequence based on the feature large kernel convolution to determine the target three - dimensional human body pose sequence, including:

[0026] Perform dimensional recombination and feature merging on the second human body pose sequence to determine the third fused feature F3 ∈ R B ×(C×J)×T ;

[0027] Use the feature large kernel convolution to learn the third fused feature to determine the third human body pose sequence, where the feature large kernel convolution is implemented based on a one - dimensional convolution with the number of groups equal to the feature dimension;

[0028] Determine the target three - dimensional human body pose sequence according to the third human body pose sequence.

[0029] In an implementation method, determining the target three - dimensional human body pose sequence according to the third human body pose sequence includes:

[0030] Superimpose the third human body pose sequence and the initial three - dimensional human body pose sequence to determine the target three - dimensional human body pose sequence.

[0031] In a second aspect, an embodiment of the present invention further provides a pose estimation device based on large kernel decoupled convolution, where the pose estimation device based on large kernel decoupled convolution includes:

[0032] A dimension conversion module, configured to obtain a two - dimensional human body pose sequence of a target user, map the two - dimensional human body pose sequence to a high - dimensional feature space, and determine an initial three - dimensional human body pose sequence F0 ∈ R B×T×J×C , where R is the set of real numbers, B is the batch size, T is the time length, J is the number of joint points, and C is the feature dimension;

[0033] A target three - dimensional human body pose determination module, configured to learn the initial three - dimensional human body pose sequence based on depth - separable decoupled convolution to determine a target three - dimensional human body pose sequence, where the depth - separable decoupled convolution is obtained by deconstructing and reconstructing a depth - separable convolution;

[0034] The three-dimensional human pose determination module is used to learn the target three-dimensional human pose sequence by using a multi-layer perceptron to determine the three-dimensional human pose of the target user.

[0035] In a third aspect, an embodiment of the present invention further provides a terminal, which includes a memory and more than one processor; the memory stores more than one program; the program includes instructions for executing the pose estimation method based on large kernel decoupled convolution as described in any one of the above; the processor is used to execute the program.

[0036] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which multiple instructions are stored, and wherein the instructions are suitable for being loaded and executed by a processor to implement the pose estimation method based on large kernel decoupled convolution as described in any one of the above.

[0037] Advantages of the present invention: In the embodiment of the present invention, by mapping the two-dimensional human pose sequence to a high-dimensional feature space, an initial three-dimensional human pose sequence F0∈R is determined, where R is the set of real numbers, B is the batch size, T is the time length, J is the number of joint points, and C is the feature dimension; based on depthwise separable decoupled convolution, the initial three-dimensional human pose sequence is learned to determine the target three-dimensional human pose sequence, and the depthwise separable decoupled convolution is obtained by deconstructing and reconstructing the depthwise separable convolution; a multi-layer perceptron is used to determine the three-dimensional human pose based on the target three-dimensional human pose sequence. The present invention uses depthwise separable decoupled convolution to independently learn the feature relationships in the three dimensions of time, space, and features, and can effectively solve the problem that the existing technology is difficult to be popularized and used in actual scenarios due to its high computational requirements and inability to estimate the human pose of the entire video sequence. B×T×J×C ,R is the set of real numbers, B is the batch size, T is the time length, J is the number of joint points, and C is the feature dimension; based on depthwise separable decoupled convolution, the initial three-dimensional human pose sequence is learned to determine the target three-dimensional human pose sequence, and the depthwise separable decoupled convolution is obtained by deconstructing and reconstructing the depthwise separable convolution; a multi-layer perceptron is used to determine the three-dimensional human pose based on the target three-dimensional human pose sequence. The present invention uses depthwise separable decoupled convolution to independently learn the feature relationships in the three dimensions of time, space, and features, and can effectively solve the problem that the existing technology is difficult to be popularized and used in actual scenarios due to its high computational requirements and inability to estimate the human pose of the entire video sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments recorded in the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0039] Figure 1 is a schematic flowchart of the pose estimation method based on large kernel decoupled convolution provided by the embodiment of the present invention.

[0040] Figure 2 is a schematic flowchart of the depthwise separable decoupled convolution provided by the embodiment of the present invention.

[0041] Figure 3 is a schematic diagram of the pose estimation result based on large kernel decoupled convolution provided by the embodiment of the present invention.

[0042] Figure 4 It is a schematic diagram of the internal modules of the pose estimation device based on large kernel decoupled convolution provided by an embodiment of the present invention.

[0043] Figure 5 It is a principle block diagram of the terminal provided by an embodiment of the present invention. Specific embodiments

[0044] The present invention discloses a pose estimation method, device, terminal and medium based on large kernel decoupled convolution. To make the objectives, technical solutions and effects of the present invention clearer and more definite, the following further describes the present invention in detail with reference to the accompanying drawings and by way of examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0045] Those skilled in the art of the present technology can understand that unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "including" used in the specification of the present invention means the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0046] Those skilled in the art of the present technology can understand that unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those of ordinary skill in the art to which the present invention belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless specifically defined as here.

[0047] Existing three-dimensional human pose estimation methods based on the attention mechanism usually regard each video frame as a pose representation and use extremely long video sequences to achieve superior performance. However, since the self-attention complexity grows quadratically with respect to the number of tokens, these methods are inevitably affected by high computational requirements, which hinders the deployment of these methods on devices with limited computational resources; while the three-dimensional human pose estimation methods based on traditional convolution can only estimate the three-dimensional human pose of the central frame of the video sequence and cannot estimate the human pose of the entire video sequence, making it equally difficult to apply to practical scenarios.

[0048] In summary, due to their high computational requirements and inability to estimate human postures for the entire video sequence, existing three-dimensional human pose estimation methods are difficult to be popularized and used in practical scenarios.

[0049] In view of the above defects of the prior art, the present invention provides a pose estimation method based on large kernel decoupled convolution. The method maps a two-dimensional human pose sequence to a high-dimensional feature space to determine an initial three-dimensional human pose sequence F0 ∈ R B×T×J×C , where R is the set of real numbers, B is the batch size, T is the time length, J is the number of joint points, and C is the feature dimension; the initial three-dimensional human pose sequence is learned based on depthwise separable decoupled convolution to determine the target three-dimensional human pose sequence, and the depthwise separable decoupled convolution is obtained by deconstructing and reconstructing depthwise separable convolution; a multi-layer perceptron is used to determine the three-dimensional human pose based on the target three-dimensional human pose sequence. The present invention uses depthwise separable decoupled convolution to independently learn the feature relationships in the three dimensions of time, space, and features, and can effectively solve the problem that existing technologies are difficult to be popularized and used in practical scenarios due to their high computational requirements and inability to estimate human postures for the entire video sequence.

[0050] Exemplary method:

[0051] As Figure 1 shown, the method includes:

[0052] Step S100, obtain the two-dimensional human pose sequence of the target user, map the two-dimensional human pose sequence to a high-dimensional feature space, and determine the initial three-dimensional human pose sequence F0 ∈ R B×T×J×C , where R is the set of real numbers, B is the batch size, T is the time length, J is the number of joint points, and C is the feature dimension.

[0053] Specifically, obtain the two-dimensional human pose sequence X ∈ R B×T×J×2 , where R is the set of real numbers, B represents the batch size, T represents the time length of the sequence, J represents the number of human joint points, and 2 represents the two-dimensional coordinates of the joint points. Use a linear layer to map the two-dimensional human pose sequence to a high-dimensional feature space to obtain the initial three-dimensional human pose sequence F0 ∈ R B ×T×J×C , where R is the set of real numbers, B is the batch size, T is the time length, J is the number of joint points, and C is the feature dimension. In this embodiment, by mapping the two-dimensional human pose sequence to a high-dimensional feature space, the semantic information of human postures can be better captured, such as the spatial relationship between joints, motion patterns, and interaction information with the scene, etc., providing a richer feature representation for the subsequent pose estimation task. This process can be described as follows:

[0054] ;

[0055] where X is a two-dimensional human body pose sequence, and F0 is an initial three-dimensional human body pose sequence, is a linear layer.

[0056] Step S200: Learn the initial three-dimensional human body pose sequence based on depthwise separable decoupled convolution to determine the target three-dimensional human body pose sequence, where the depthwise separable decoupled convolution is obtained by deconstructing and reconstructing a depthwise separable convolution.

[0057] Briefly, in traditional depthwise separable convolution, when jointly learning different dimensional dependencies, information in the time, space, and feature dimensions is mixed. This strategy often leads to a significant decline in feature expression ability and insufficient utilization of information. In this embodiment, the traditional depthwise separable convolutional structure is deconstructed and reconstructed to obtain depthwise separable decoupled convolution, and the initial three-dimensional human body pose sequence is learned based on the depthwise separable decoupled convolution, enabling it to independently learn the feature relationships in the time, space, and feature dimensions. The convolution in each dimension focuses on learning the feature relationships in its specific dimension, avoiding improper mixing of information.

[0058] In one implementation, the depthwise separable decoupled convolution includes a temporal large kernel convolution, a spatial large kernel convolution, and a feature large kernel convolution. Learning the initial three-dimensional human body pose sequence based on the depthwise separable decoupled convolution to determine the target three-dimensional human body pose sequence includes:

[0059] Step S201: Learn the initial three-dimensional human body pose sequence based on the temporal large kernel convolution to determine the first human body pose sequence;

[0060] Step S202: Learn the first human body pose sequence based on the spatial large kernel convolution to determine the second human body pose sequence;

[0061] Step S203: Learn the second human body pose sequence based on the feature large kernel convolution to determine the target three-dimensional human body pose sequence.

[0062] Specifically, as Figure 2As shown, the depth - separable decoupled convolution can be split into three convolutional components: independently, the temporal large - kernel convolution, the spatial large - kernel convolution, and the feature large - kernel convolution. The temporal large - kernel convolution, the spatial large - kernel convolution, and the feature large - kernel convolution are respectively used to learn the feature relationships in the temporal, spatial, and feature dimensions of the initial three - dimensional human pose sequence. In addition, different from traditional small convolution kernels (usually 3×3 or 5×5), in this embodiment, an enlarged convolution kernel size is also introduced in the depth - separable decoupled convolution, which expands the receptive field of the convolution and improves the global perception ability of feature extraction. By introducing large - kernel convolutions with large convolution kernel sizes in the depth - separable decoupled convolution, through fine - grained dimension decoupling and receptive field expansion, it can capture the spatio - temporal features in the human pose sequence more accurately and comprehensively, while effectively improving the feature expression ability and maintaining the computational efficiency and generalization performance.

[0063] In one implementation, learning the initial three - dimensional human pose sequence based on the temporal large - kernel convolution to determine the first human pose sequence includes:

[0064] Step S2011: Reorganize the dimensions and merge the features of the initial three - dimensional human pose sequence to determine the first fusion feature F1∈R B×(J×C)×T ;

[0065] Step S2012: Perform reflection padding on the first fusion feature based on the convolution kernel size of the temporal large - kernel convolution;

[0066] Step S2013: Use the temporal large - kernel convolution to learn the reflected - padded first fusion feature to determine the first human pose sequence.

[0067] The temporal large - kernel convolution is used to extract the variation law of each joint of the initial three - dimensional human pose sequence in the temporal dimension. The temporal large - kernel convolution can be regarded as a sliding window. This sliding window slides along the time axis. Each time, the features within the sliding window are used to calculate a new feature, capturing local temporal dependencies, extracting the sequence features in the temporal dimension, and perceiving the correlation between different time steps through the convolution kernel, thereby extracting the motion trend of each joint over a period of time.

[0068] Specifically, before using the temporal large - kernel convolution to learn the initial three - dimensional human pose sequence, first reorganize the dimensions and merge the features of the initial three - dimensional human pose sequence. The initial three - dimensional human pose sequence F0∈R B×T×J×CThe structure is represented as [B, T, J, C]. The initial three-dimensional human pose sequence is dimensionally reorganized so that its structure changes to [B, J, C, T]. Feature merging is performed on the reorganized initial three-dimensional human pose sequence, combining the number of joint points and the feature dimensions of the initial three-dimensional human pose sequence, with the time length as the convolution dimension, to obtain the first fusion feature F1 ∈ R with the structure [B, J×C, T] B ×(J×C)×T , preparing for subsequent one-dimensional convolution.

[0069] Reflection padding is performed on the first fusion feature. The specific method includes: calculating the reflection values for padding on both sides of the time dimension based on the convolution kernel size self.kernel_size of the time large kernel convolution. The pseudocode for calculating the reflection values is as follows:

[0070] pad_left = self.kernel_size / / 2;

[0071] pad_right = self.kernel_size / / 2;

[0072] where self is the parameter name, pad_left is the left reflection value, pad_right is the right reflection value, and / / represents integer division with rounding down.

[0073] The first fusion feature is padded with the calculated reflection values, padding kernel_size / / 2 reflection values on both sides of the time dimension, avoiding the loss of head and tail frame information during convolution and maintaining motion continuity.

[0074] The time large kernel convolution is used to learn the reflected padded first fusion feature. Convolution is performed independently for each (J×C) channel. The convolution kernel slides along the time axis to capture local motion patterns and determine the first human pose sequence:

[0075] x = self.dw_conv(x);

[0076] where dw_conv is the component of the time large kernel convolution of the depthwise separable decoupled convolution, and x in self.dw_conv(x) is the first fusion feature. After learning by the time large kernel convolution, the first human pose sequence x is obtained (x is a variable in the pseudocode, and the data represented by x changes according to different processes).

[0077] In one implementation, the time large kernel convolution is implemented based on a one-dimensional convolution with a convolution kernel size of 17×17 and a number of groups equal to the product of the number of joint points and the feature dimensions.

[0078] Briefly speaking, this embodiment uses a one-dimensional convolution with a kernel size of 17×17 to implement a large temporal kernel convolution, and sets the number of groups (groups) of this one-dimensional convolution to the product of the number of joint points and the feature dimension (J×C), so that the motion trend of human joints over a period of time can be captured from the initial three-dimensional human pose sequence based on the large temporal kernel convolution.

[0079] The pseudo-code representation of setting the large temporal kernel convolution is as follows:

[0080] self.kernel_size = kernel_size;

[0081] self.dw_conv = nn.Conv1d(in_channels = J*C, out_channels = J*C, kernel_size = kernel_size, groups = J*C).

[0082] Among them, self is the parameter name, kernel_size is the kernel size, dw_conv is the component of the large temporal kernel convolution of the depthwise separable decoupled convolution, nn.Conv1d() is the class for constructing one-dimensional convolution, in_channels is the number of input channels, out_channels is the number of output channels, groups is the number of groups between the input channels and the output channels. In this embodiment, the kernel size kernel_size is set to 17×17, and * represents the multiplication operation.

[0083] In one implementation, learning the first human pose sequence based on the large spatial kernel convolution to determine the second human pose sequence includes:

[0084] Step S2021, reducing the dimension of the first human pose sequence so that the dimension of the first human pose sequence is the same as that of the initial three-dimensional human pose sequence;

[0085] Step S2022, normalizing, dimension restructuring, and feature merging the dimension-reduced first human pose sequence to determine the second fusion feature F2∈R B×(J×C)×T ;

[0086] Step S2023, using the large spatial kernel convolution to learn the second fusion feature to determine the second human pose sequence, where the large spatial kernel convolution is implemented based on a one-dimensional convolution with the number of groups being the number of joint points.

[0087] The large spatial kernel convolution is used to extract the variation law in the spatial dimension, can capture the spatial relationship between different human joints, capture local spatial features, learn the spatial topological structure, and extract the hierarchical representation of spatial features, such as the relative positions of the arms and legs.

[0088] Specifically, the dimensionality of the first human body pose sequence is restored to the original [B, T, J, C] structure, and the dimensionality-restored first human body pose sequence is normalized, with the feature vectors of each joint point being independently normalized to stabilize the training process. The normalized first human body pose sequence is subjected to dimensionality recombination and feature merging, and the structure is converted from the [B, T, J, C] structure to the [B, (J×C), T] structure, obtaining the second fusion feature F2 ∈ R B×(J×C)×T Spatial large kernel convolution is used to learn the second fusion feature, performing a spatial interaction branch, grouping by the number of joint points, and learning the spatial correlation between joints to obtain the second human body pose sequence:

[0089] x = self.spatial_convffn(x);

[0090] where spatial_convffn() is the spatial large kernel convolution, and x in self.spatial_convffn(x) is the second fusion feature. After learning by the spatial large kernel convolution, the second human body pose sequence x is obtained.

[0091] In one implementation, the second human body pose sequence is learned based on the feature large kernel convolution to determine the target three-dimensional human body pose sequence, including:

[0092] Step S2031: Perform dimensionality recombination and feature merging on the second human body pose sequence to determine the third fusion feature F3 ∈ R B×(C×J)×T ;

[0093] Step S2032: Use the feature large kernel convolution to learn the third fusion feature to determine the third human body pose sequence, where the feature large kernel convolution is implemented based on one-dimensional convolution with the number of groups being the feature dimension;

[0094] Step S2033: Determine the target three-dimensional human body pose sequence according to the third human body pose sequence.

[0095] The feature large kernel convolution is used to learn the feature dimension, extracting more abstract and discriminative features, such as non-linear transformation of the feature space, cross-channel feature interaction, abstraction and compression of features, such as human motion patterns, pose categories, etc.

[0096] Specifically, the dimensionality of the second human body pose sequence is recombined, and the structure of the second human body pose sequence is converted to the [B, C, J, T] structure, and the dimensionality-recombined second human body pose sequence is subjected to feature merging to obtain the third fusion feature F3 ∈ R with the structure of [B, C×J, T] B×(C×J)×TThe third fused feature is learned using a large kernel convolution of features to implement a feature interaction branch. It is grouped by feature dimension to mine cross-channel feature combinations (e.g., the coupling relationship between rotation angle and speed), and the third human pose sequence is obtained:

[0097] x = self.feature_convffn(x);

[0098] Among them, feature_convffn() is the large kernel convolution of features. In self.feature_convffn(x), x is the third fused feature. After being learned by the large kernel convolution of features, the third human pose sequence x is obtained.

[0099] In one implementation, the method for implementing the spatial large kernel convolution based on a one-dimensional convolution with the number of groups equal to the number of joint points and the feature large kernel convolution based on a one-dimensional convolution with the number of groups equal to the feature dimension is as follows:

[0100] self.spatial_convffn = ConvFFN(J, C, feature_ratio, mode='spatial');

[0101] self.feature_convffn = ConvFFN(J, C, feature_ratio, mode='feature');

[0102] Among them, feature_ratio is the feature ratio, that is, the proportion of the number of features to be retained in the total number of features; mode is a model parameter. mode='spatial' indicates constructing a spatial large kernel convolution, and mode='feature' indicates constructing a feature large kernel convolution. ConvFFN() is a method for constructing a spatial large kernel convolution or a feature large kernel convolution. An example for implementing the function of ConvFFN() is as follows:

[0103] group_num = J if mode =='spatial' else C (when the model parameter mode of ConvFFN() is'spatial', let the number of groups group_num be the number of joint points, otherwise let the number of groups group_num be the feature dimension);

[0104] self.pw_conv1 = nn.Conv1d(in_channels=C*J, out_channels=feature_ratio*C*J, kernel_size=1, groups=group_num) (Create a one-dimensional convolutional layer using nn.Conv1d(). The number of input channels of this one-dimensional convolutional layer is the product of the feature dimension and the number of joint points, the number of output channels is the product of the feature ratio, the feature dimension, and the number of joint points, the convolutional kernel size is 1×1, and the number of groups is the value assigned in the previous step);

[0105] self.act = nn.GELU() (self.act is an instance attribute, and nn.GELU() is an activation function. Use the activation function to improve the performance and generalization ability of the ConvFFN() method);

[0106] self.pw_conv2 = nn.Conv1d(in_channels=feature_ratio*C*J, out_channels=C*J, kernel_size=1, groups=group_num) (Create a one-dimensional convolutional layer using nn.Conv1d(). The number of input channels of this one-dimensional convolutional layer is the product of the feature ratio, the feature dimension, and the number of joint points, the number of output channels is the product of the feature dimension and the number of joint points, the convolutional kernel size is 1×1, and the number of groups is the value assigned in the previous step).

[0107] In one implementation, determining the target three-dimensional human pose sequence according to the third human pose sequence includes:

[0108] Superimpose the third human pose sequence and the initial three-dimensional human pose sequence to determine the target three-dimensional human pose sequence.

[0109] Simply put, residually superimpose the third human pose sequence and the initial three-dimensional human pose sequence, fuse the original features and the learning increment, and alleviate the vanishing gradient. Before residually superimposing the third human pose sequence and the initial three-dimensional human pose sequence, first reduce the dimension of the third human pose sequence to make its dimension the same as that of the initial three-dimensional human pose sequence, so as to facilitate the subsequent residual superimposition operation.

[0110] Step S300: Use a multi-layer perceptron to learn the target three-dimensional human pose sequence to determine the three-dimensional human pose of the target user.

[0111] Specifically, learn the target three-dimensional human pose sequence through a multi-layer perceptron to obtain the final three-dimensional human pose sequence Y∈R B×T×J×3 , so as to obtain the three-dimensional human pose of the target user. The specific representation is as follows:

[0112] ;

[0113] Among them, is a multi-layer perceptron, and F N is the target three-dimensional human body pose sequence.

[0114] As Figure 3 shown, Figure 3 in Figure (a) is the input data, and Figure (b) is the output three-dimensional human body pose. Based on the above embodiments, the three-dimensional human body pose under occlusion and rapid movement can be accurately estimated, and it has excellent generalization ability.

[0115] Based on the above embodiments, the present invention also provides a pose estimation device based on large kernel decoupled convolution. As Figure 4 shown, the device includes:

[0116] A dimension conversion module 01, configured to obtain the two-dimensional human body pose sequence of the target user, map the two-dimensional human body pose sequence to a high-dimensional feature space, and determine an initial three-dimensional human body pose sequence F0 ∈ R B×T×J×C , where R is the set of real numbers, B is the batch size, T is the time length, J is the number of joint points, and C is the feature dimension;

[0117] A target three-dimensional human body pose determination module 02, configured to learn the initial three-dimensional human body pose sequence based on depthwise separable decoupled convolution to determine the target three-dimensional human body pose sequence, where the depthwise separable decoupled convolution is obtained by deconstructing and reconstructing the depthwise separable convolution;

[0118] A three-dimensional human body pose determination module 03, configured to learn the target three-dimensional human body pose sequence by using a multi-layer perceptron to determine the three-dimensional human body pose of the target user.

[0119] Based on the above embodiments, the present invention also provides a terminal, and its principle block diagram can be as Figure 5 shown. The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal through a network connection. The computer program, when executed by the processor, implements a pose estimation method based on large kernel decoupled convolution. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0120] Those skilled in the art can understand, Figure 5The principle block diagram shown only shows the block diagram of some structures related to the solution of the present invention, and does not constitute a limitation on the terminal to which the solution of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0121] In one implementation, more than one program is stored in the memory of the terminal, and is configured to be executed by more than one processor. The more than one program includes instructions for performing the pose estimation method based on large kernel decoupled convolution.

[0122] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided by the present invention can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0123] In summary, the present invention discloses a pose estimation method, device, terminal, and medium based on large kernel decoupled convolution. The method maps a two-dimensional human pose sequence to a high-dimensional feature space to determine an initial three-dimensional human pose sequence F0 ∈ R B×T×J×C, where \(R\) is the set of real numbers, \(B\) is the batch size, \(T\) is the time length, \(J\) is the number of joint points, and \(C\) is the feature dimension; learning the initial three-dimensional human pose sequence based on depthwise separable decoupled convolution to determine the target three-dimensional human pose sequence, and the depthwise separable decoupled convolution is obtained by deconstructing and reconstructing the depthwise separable convolution; using a multi-layer perceptron to determine the three-dimensional human pose based on the target three-dimensional human pose sequence. The present invention uses depthwise separable decoupled convolution to independently learn the feature relationships in the three dimensions of time, space, and features, and can effectively solve the problem that the existing technology is difficult to be popularized and used in actual scenarios due to its high computational requirements and inability to estimate the human pose of the entire video sequence.

[0124] It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or transformations can be made according to the above description, and all such improvements and transformations should fall within the protection scope of the appended claims of the present invention.

Claims

1. A pose estimation method based on large kernel decoupled convolution, characterized in that, The method includes: Obtain the two-dimensional human body pose sequence of the target user, map the two-dimensional human body pose sequence to a high-dimensional feature space, and determine the initial three-dimensional human body pose sequence F0∈R B×T×J×C , where R is the set of real numbers, B is the batch size, T is the time length, J is the number of joint points, and C is the feature dimension; Learning the initial three-dimensional human pose sequence based on depthwise separable decoupled convolution to determine the target three-dimensional human pose sequence, where the depthwise separable decoupled convolution is obtained by deconstructing and reconstructing the depthwise separable convolution; Using a multi-layer perceptron to learn the target three-dimensional human pose sequence to determine the three-dimensional human pose of the target user; The depthwise separable decoupled convolution includes a temporal large kernel convolution, a spatial large kernel convolution, and a feature large kernel convolution. Learning the initial three-dimensional human pose sequence based on the depthwise separable decoupled convolution to determine the target three-dimensional human pose sequence includes: Learning the initial three-dimensional human pose sequence based on the temporal large kernel convolution to determine the first human pose sequence; Learning the first human pose sequence based on the spatial large kernel convolution to determine the second human pose sequence; Learning the second human pose sequence based on the feature large kernel convolution to determine the target three-dimensional human pose sequence; Learning the initial three-dimensional human pose sequence based on the temporal large kernel convolution to determine the first human pose sequence, including: Perform dimensional reorganization and feature merging on the initial three-dimensional human body pose sequence to determine the first fusion feature F1 ∈ R B ×(J×C)×T ; Performing reflection padding on the first fused feature based on the convolution kernel size of the temporal large kernel convolution; Using the temporal large kernel convolution to learn the reflected and padded first fused feature to determine the first human pose sequence; Perform dimensional recombination and feature merging on the initial three-dimensional human body pose sequence to determine the first fusion feature F1 ∈ R B ×(J×C)×T , including: The structure of the initial three-dimensional human pose sequence is represented as [B, T, J, C]. Reorganizing the dimensions of the initial three-dimensional human pose sequence changes the structure to [B, J, C, T]; Perform feature merging on the initial three-dimensional human pose sequence after dimension reorganization, merge the number of joint points and feature dimensions of the initial three-dimensional human pose sequence, and use the time length as the convolution dimension to obtain the first fusion feature F1 ∈ R with the structure [B, J×C, T] B×(J×C)×T .

2. The pose estimation method based on large-core decoupled convolution according to claim 1, wherein The temporal large kernel convolution is implemented based on a one-dimensional convolution with a convolution kernel size of 17×17 and a number of groups equal to the product of the number of joint points and the feature dimension.

3. The pose estimation method based on large-core decoupled convolution according to claim 1, characterized in that Learning the first human pose sequence based on the spatial large kernel convolution to determine the second human pose sequence, including: Restoring the dimensions of the first human pose sequence so that the first human pose sequence has the same dimensions as the initial three-dimensional human pose sequence; Normalize, dimensionally reorganize, and feature merge the first human body pose sequence after dimension reduction to determine the second fused feature F2 ∈ R B×(J×C)×T ; Using the spatial large kernel convolution to learn the second fused feature to determine the second human pose sequence, where the spatial large kernel convolution is implemented based on a one-dimensional convolution with a number of groups equal to the number of joint points.

4. The pose estimation method based on large-core decoupled convolution according to claim 1, characterized in that, Learning the second human pose sequence based on the feature large kernel convolution to determine the target three-dimensional human pose sequence, including: Perform dimensional recombination and feature merging on the second human body pose sequence to determine the third fusion feature F3 ∈ R B×(C×J)×T ; Using the feature large kernel convolution to learn the third fused feature to determine the third human pose sequence, where the feature large kernel convolution is implemented based on a one-dimensional convolution with a number of groups equal to the feature dimension; Determining the target three-dimensional human pose sequence according to the third human pose sequence.

5. The pose estimation method based on large-core decoupled convolution according to claim 4, wherein Determining the target three-dimensional human pose sequence according to the third human pose sequence, including: Overlaying the third human pose sequence with the initial three-dimensional human pose sequence to determine the target three-dimensional human pose sequence.

6. An attitude estimation device based on large-core decoupled convolution, characterized in that, The device includes: A dimension conversion module, which is used to obtain a two-dimensional human body pose sequence of a target user, map the two-dimensional human body pose sequence to a high-dimensional feature space, and determine an initial three-dimensional human body pose sequence F0 ∈ R B×T×J×C , where R is the set of real numbers, B is the batch size, T is the time length, J is the number of joint points, and C is the feature dimension; A target three-dimensional human pose determination module is used to learn the initial three-dimensional human pose sequence based on depthwise separable decoupled convolution to determine a target three-dimensional human pose sequence, where the depthwise separable decoupled convolution is obtained by deconstructing and reconstructing a depthwise separable convolution; A three-dimensional human pose determination module is used to learn the target three-dimensional human pose sequence by using a multi-layer perceptron to determine the three-dimensional human pose of the target user; The depthwise separable decoupled convolution includes a temporal large kernel convolution, a spatial large kernel convolution, and a feature large kernel convolution. Learning the initial three-dimensional human pose sequence based on the depthwise separable decoupled convolution to determine a target three-dimensional human pose sequence includes: Learning the initial three-dimensional human pose sequence based on the temporal large kernel convolution to determine a first human pose sequence; Learning the first human pose sequence based on the spatial large kernel convolution to determine a second human pose sequence; Learning the second human pose sequence based on the feature large kernel convolution to determine the target three-dimensional human pose sequence; Learning the initial three-dimensional human pose sequence based on the temporal large kernel convolution to determine a first human pose sequence, including: Perform dimensional recombination and feature merging on the initial three-dimensional human body pose sequence to determine the first fusion feature F1 ∈ R B ×(J×C)×T ; Performing reflection padding on the first fused feature based on the convolutional kernel size of the temporal large kernel convolution; Using the temporal large kernel convolution to learn the reflected and padded first fused feature to determine the first human pose sequence; Perform dimensional recombination and feature merging on the initial three-dimensional human body pose sequence to determine the first fusion feature F1 ∈ R B ×(J×C)×T , including: The structure of the initial three-dimensional human pose sequence is represented as [B, T, J, C]. The dimension of the initial three-dimensional human pose sequence is reorganized, and the structure is changed to [B, J, C, T]; Perform feature merging on the initial three-dimensional human pose sequence after dimension reorganization, merge the number of joint points and feature dimensions of the initial three-dimensional human pose sequence, use the time length as the convolutional dimension, and obtain the first fusion feature F1 ∈ R with the structure of [B, J×C, T] B×(J×C)×T .

7. A terminal, characterized in that, The terminal includes a memory and more than one processor; the memory stores more than one program; the program includes instructions for executing the pose estimation method based on large kernel decoupled convolution as described in any one of claims 1-5; the processor is used to execute the program.

8. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that, The instructions are loaded and executed by the processor to implement the steps of the pose estimation method based on large kernel decoupled convolution as described in any one of claims 1-5 above.

Images