A method and apparatus for 3D human pose estimation using lightweight convolutional networks

Abstract

This invention presents a lightweight convolutional network method for 3D human pose estimation. It employs a multi-layered progressive backbone network composed of multiple depthwise separable dilated residual convolutional modules connected in series. Multi-stage refined supervision is achieved using upscaling modules of different scales and depthwise separable dilated residual convolutional modules. The method takes a 2D human joint sequence as input and outputs a 3D human coordinate sequence corresponding to the input 2D joint coordinates. This invention utilizes depthwise separable dilated residual convolutional modules and multi-stage refined feature extraction to achieve the mapping from 2D to 3D human pose, thus solving the depth ambiguity and ill-posedness problems inherent in 3D human pose estimation tasks and improving estimation accuracy.

Description

Technical Field

[0001] This invention belongs to the field of 3D human pose estimation, and in particular relates to a 3D human pose estimation method based on a lightweight convolutional network with multi-stage refined supervision. Background Technology

[0002] 3D human pose estimation is a hot research topic in computer vision, serving as a foundation for research in multiple fields such as autonomous driving, motion-sensing games, intelligent monitoring, and human-computer interaction. In recent years, deep learning-based 3D human pose estimation methods can be broadly categorized into two types: end-to-end methods and staged methods. End-to-end methods estimate the coordinates of the 3D human pose from RGB images, directly predicting and making better use of information from the original image. Staged methods divide the task into two stages: the first stage estimates the 2D human pose from video or images, and the second stage uses the 2D human pose as input to estimate the 3D human pose, primarily relying on the accuracy of the 2D pose estimation. Because 2D pose estimation is more mature and this type of method is more universal than end-to-end methods, staged methods are more mainstream.

[0003] However, these methods face a significant challenge due to the inherent depth ambiguity and ill-posedness in the mapping from 2D to 3D pose, resulting in low accuracy for 3D human pose estimators when dealing with complex movements. Currently, addressing the depth ambiguity and ill-posedness issues inherent in 3D human pose estimation and improving estimation accuracy are problems that need to be solved.

[0004] Purpose of the invention

[0005] To address the shortcomings of existing technologies, this invention provides a lightweight convolutional network-based method for 3D human pose estimation. It employs a depth-separable dilated residual convolutional module and multi-stage refined feature extraction to map 2D human pose to 3D human pose, thus solving the depth ambiguity and ill-posedness problems in 3D human pose estimation tasks and improving estimation accuracy.

[0006] This invention provides a method for 3D human pose estimation using lightweight convolutional networks. It employs multiple depth-separable dilated residual convolutional modules connected in series to form a multi-layered progressive backbone network. It utilizes dimensionality-upgrading modules of different scales and depth-separable dilated residual convolutional modules to form multi-stage refined supervision. The method takes a 2D human joint sequence as input and outputs a 3D human coordinate sequence corresponding to the input 2D joint coordinates.

[0007] Further steps include the following:

[0008] Step 1: Extract the two-dimensional human joint coordinate sequence. Assuming the given two-dimensional human joint coordinate sequence is 243 frames, the input two-dimensional human joint coordinate sequence is processed by a dimension-up convolution module at different scales. First, the dimension-up convolution module at different scales is executed. This dimension-up convolution module consists of a slicing function, a one-dimensional convolution with a kernel of 3, a stride of 3, and 1024 channels, batch normalization, Mish activation function, and random deactivation. After processing the input two-dimensional human joint coordinate sequence at different scales, the processing results with feature scales of 81*1024, 27*1024, 9*1024, and 3*1024 are output respectively.

[0009] Step 2: Using residual connections, the output of the upscaling results at different scales is fused into each layer of a four-layer progressive network structure composed of multiple layers of depth-separable dilated residual convolutional modules connected in series, forming multi-stage supervision. The model depth and feature input are enhanced by constructing a multi-layer progressive layer structure through multiple layers of depth-separable dilated residual convolutional modules connected in series.

[0010] In the depthwise separable dilated residual convolution module, the depthwise separable dilated residual convolution is first executed. This convolution consists of a point convolution module composed of a one-dimensional convolution with a kernel of 1, batch normalization, and the Mish activation function, and a convolution kernel with a kernel of w and a dilation factor of d = w. b The dilated channel-wise convolution module, consisting of channel-wise convolution, batch normalization, Mish activation function, and random deactivation, is executed. A slicing function is used to perform residual connection between the point convolution module and the dilated channel-wise convolution module to achieve depth-separable dilated residual convolution. The slicing function obtains information features that match the dimensions of the convolution before and after, which avoids overfitting and prevents network degradation. Then, a point convolution module and a random deactivation function are executed to fuse the information of each channel of the convolution and output the result.

[0011] Step 3: To further explore the feature value of shallow networks, the shallow features output by the first-layer upscaling convolutional module and the first-layer depthwise separable dilated residual convolutional module are respectively fed into the depthwise separable strided residual convolutional module for dimensionality reduction. The shallow features are further optimized and refined to output deep features. The output of the first-layer upscaling convolutional module, after dimensionality reduction by the depthwise separable strided residual convolutional module, is input into the third and fourth-layer depthwise separable dilated residual convolutional modules. The output of the first-layer depthwise separable dilated residual convolutional module, after dimensionality reduction by the depthwise separable strided residual convolutional module, is then input into the fourth-layer depthwise separable dilated residual convolutional module. The residual connections connect the shallow feature output and the deep feature input to achieve refined supervision.

[0012] The difference between this depthwise separable strided residual convolution module and the depthwise separable dilated residual convolution module lies in the dilation factor setting and stride setting; the depthwise separable strided residual convolution module does not use dilated convolution, and the stride is set to 3 and the dilation factor is 1.

[0013] Step 4: The output of the fourth layer depth-separable dilated residual convolution module is fed to the fully connected layer. All feature information accumulated by the network is fused through the convolution of the fully connected layer, and the number of channels is reduced from 1024 to 51. Finally, a 1*3*17 three-dimensional human pose estimation result is output.

[0014] The process of upscaling the input two-dimensional human joint coordinate sequence at different scales is specifically expressed as follows:

[0015] F1 k =SLICE k (X k )+ACN k (X k )

[0016] Among them, X k Represents a two-dimensional human joint sequence, ACN k SLICE represents the up-dimensional convolution module. k This represents the slice function module, where k represents the k-th dimensionality increase process.

[0017] The depth-separable void residual module extracts the posture representation as follows:

[0018]

[0019] Where, X k Represents a two-dimensional human pose sequence, DSN k represents a depthwise separable dilated residual convolutional module, and k represents the k-th depthwise separable dilated residual convolutional module.

[0020] The convolutional fusion in step 4 is represented as:

[0021] F = F1 + F2 + F3

[0022] Here, F1 represents the dimension-upgrading convolution module, F2 represents the depth-separable dilated residual convolution module, and F3 represents the depth-separable strided residual convolution module.

[0023] A computer device includes a processor and a memory, the memory storing a computer program that, when executed by the processor, implements a method for 3D human pose estimation using any of the aforementioned lightweight convolutional networks.

[0024] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for 3D human pose estimation using any of the aforementioned lightweight convolutional networks.

[0025] This invention employs a depth-separable dilated residual convolution module for the field of 3D human pose estimation. This module comprises a point convolution module consisting of one-dimensional convolution, batch normalization, and activation functions, and a convolution kernel with kernel w and dilation factor d = w. b It consists of a dilated channel-wise convolution module composed of channel-wise convolution, batch normalization, Mish activation function, and random deactivation, and introduces residual connections between the point convolution module and the dilated channel-wise convolution module. This convolution module can greatly reduce the number of parameters and computational complexity while improving model performance.

[0026] This invention employs an upscaling convolution module, consisting of a slicing function, one-dimensional convolution, batch normalization, Mish activation function, and random deactivation, to perform upscaling processing on the model input features at different scales. Residual connections are used to form multi-stage supervision. Furthermore, a depthwise separable strided convolution module is used to further optimize the shallow network through convolution, and the slicing function is used for residual connections to achieve refined supervision. Combining shallow and deep features improves the generalization ability of the neural network and effectively enhances gradient propagation.

[0027] This invention employs a depthwise separable strided residual convolution module to process shallow features, further refining and optimizing them to obtain more accurate features for propagation to deeper layers. In experiments, the use of the depthwise separable strided residual convolution module for further feature extraction and fusion effectively improved the model accuracy by 1.3% compared to the original model.

[0028] Compared with existing technologies, the 3D human pose estimation method of the lightweight convolutional network of this invention has the following advantages: (1) The use of lightweight convolution in the model structure can greatly reduce the number of parameters and computational complexity compared with traditional convolution; (2) The use of the up-dimensional convolution module and the depth-separable strided residual convolution module to form multi-stage supervision and fine supervision by using multi-level residual connections respectively, improves the overall perception capability from coarse to fine in the model structure, makes full use of the feature information of each level of the network, effectively improves the model accuracy by 2.6% on the original basis, and makes the human pose smoother and more coherent; (3) The proposed depth-separable dilated residual convolution module can effectively reduce the number of parameters and computational complexity while also effectively improving the model performance. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the architecture of the three-dimensional human pose estimation network of the present invention;

[0030] Figure 2This is a schematic diagram of the depth-separable holed residual convolution module in this invention;

[0031] Figure 3 This is a schematic diagram of the depth-separable strided residual convolution module in this invention;

[0032] Figure 4 This is a schematic diagram of the dimension-upgrading convolution module in this invention;

[0033] Figure 5 This is the overall flowchart of the present invention.

[0034] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Detailed Implementation

[0035] like Figure 1-5 As shown, Embodiment 1 of the present invention provides a method for 3D human pose estimation using a lightweight convolutional network. It employs multiple depthwise separable dilated residual convolutional modules connected in series to form a multi-layered progressive backbone network. The input 2D human joint coordinate sequence is processed by an upscaling module, a depthwise separable dilated residual convolutional module, and a depthwise separable striding convolutional module. Finally, a fusion convolution is performed to output a 3D human coordinate sequence corresponding to the input 2D joint coordinates. Specifically, the method includes the following steps:

[0036] Step 1: Extract the two-dimensional human joint coordinate sequence using the CPN two-dimensional human detector. Assuming the given two-dimensional human joint coordinate sequence is 243 frames, perform dimensionality upscaling processing on the input two-dimensional human joint coordinate sequence at different scales using the dimensionality upscaling convolution module:

[0037] First, different scales of dimensionality-upgrading convolution modules are executed. These modules consist of a slicing function, a one-dimensional convolution with a kernel of 3, a stride of 3, and 1024 channels, batch normalization, the Mish activation function, and random deactivation. After performing dimensionality-upgrading processing on the input human two-dimensional joint coordinate sequence at different scales, the processing results with feature scales of 81*1024, 27*1024, 9*1024, and 3*1024 are output respectively.

[0038] The process of upscaling the input two-dimensional human joint coordinate sequence at different scales is specifically expressed as follows:

[0039] F1 k =SLICE k (X k )+ACN k (X k )

[0040] Among them, X k Represents a two-dimensional human joint sequence, ACN k SLICE represents the up-dimensional convolution module.k This represents the slice function module, where k represents the k-th dimension upscaling process;

[0041] Step 2: Using residual connections, the output of the dimensionality-upgraded results at different scales are respectively fused into each layer of a four-layer progressive network structure composed of multiple layers of depth-separable dilated residual convolutional modules connected in series, forming multi-stage supervision. The model depth is enhanced by constructing a multi-layer progressive layer structure through multiple layers of depth-separable dilated residual convolutional modules connected in series.

[0042] In the depthwise separable dilated residual convolution module, the depthwise separable dilated residual convolution is first executed. This convolution consists of a point convolution module composed of a one-dimensional convolution with a kernel of 1, batch normalization, and the Mish activation function, and a convolution kernel with a kernel of w and a dilation factor of d = w. b The dilated channel-wise convolution module, consisting of channel-wise convolution, batch normalization, Mish activation function, and random deactivation, is executed. A slicing function is used to perform residual connection between the point convolution module and the dilated channel-wise convolution module to achieve depth-separable dilated residual convolution. The slicing function obtains information features that match the dimensions of the convolution before and after, which avoids overfitting and prevents network degradation. Then, a point convolution module and a random deactivation function are executed to fuse the information of each channel of the convolution and output the result.

[0043] In form, the pose extracted by the depth-separable void residual module for human pose can be represented as:

[0044]

[0045] Where, X k Represents a two-dimensional human pose sequence, DSN k This represents a depth-separable dilated residual convolutional module, where k represents the k-th depth-separable dilated residual convolutional module.

[0046] Step 3: To further explore the feature value of shallow networks, the shallow features output by the first-layer upscaling convolutional module and the first-layer depthwise separable dilated residual convolutional module are respectively input into the depthwise separable strided residual convolutional module for dimensionality reduction processing, and the shallow features are further optimized and refined to output deep features. The output of the first-layer upscaling convolutional module is output after dimensionality reduction processing by the depthwise separable strided residual convolutional module, and then input into the third and fourth-layer depthwise separable dilated residual convolutional modules respectively. The output of the first-layer depthwise separable dilated residual convolutional module is output after dimensionality reduction processing by the depthwise separable strided residual convolutional module, and then input into the fourth-layer depthwise separable dilated residual convolutional module. The residual connections connect the shallow feature output and the deep feature input to achieve refined supervision.

[0047] The difference between this depthwise separable strided residual convolution module and the depthwise separable dilated residual convolution module lies in the dilation factor setting and stride setting; the depthwise separable strided residual convolution module does not use dilated convolution to avoid unnecessary calculations, and the stride is set to 3 and the dilation factor to 1.

[0048] In form, the three-dimensional human pose estimation model can be represented as:

[0049]

[0050] Among them, X k DSSN represents a two-dimensional human joint sequence. k represents a depthwise separable strided residual convolutional module, and k represents the k-th depthwise separable dilated residual convolutional module.

[0051] Step 4: The output of the fourth layer depth-separable dilated residual convolution module is fed to the fully connected layer. All feature information accumulated by the network is fused through the convolution of the fully connected layer, and the number of channels is reduced from 1024 to 51. Finally, a 1*3*17 three-dimensional human pose estimation result is output.

[0052] The convolutional fusion can be represented as:

[0053] F = F1 + F2 + F3

[0054] Here, F1 represents the dimension-upgrading convolution module, F2 represents the depth-separable dilated residual convolution module, and F3 represents the depth-separable strided residual convolution module.

[0055] In Embodiment 2, the present invention provides a computer device including a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. The processor provides computational and control capabilities, and the memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. When the computer program is executed by the processor, it implements the method for 3D human pose estimation using a lightweight convolutional network as described in Embodiment 1. Embodiments 1 and 2 have the same functionality and beneficial effects.

[0056] In Embodiment 3, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the method for 3D human pose estimation using a lightweight convolutional network as described in Embodiment 1. Embodiments 1 and 3 have the same functionality and beneficial effects.

[0057] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. The non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0058] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0059] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for 3D human pose estimation using lightweight convolutional networks, characterized in that: A multi-layered progressive backbone network is constructed by cascading multiple depthwise separable dilated residual convolutional modules. Multi-stage refined supervision is achieved using dimensionality-upgrading modules of different scales and depthwise separable dilated residual convolutional modules. The network takes a 2D human joint sequence as input and outputs a 3D human coordinate sequence corresponding to the input 2D joint coordinates. The process includes the following steps: Step 1: Assuming a given 243 frames of two-dimensional human joint coordinate sequence, the input two-dimensional human joint coordinate sequence is processed by a dimension-up convolution module at different scales. First, the dimension-up convolution module at different scales is executed. This dimension-up convolution module consists of a slicing function, a one-dimensional convolution with a kernel of 3, a stride of 3, and 1024 channels, batch normalization, Mish activation function, and random deactivation. After processing the input two-dimensional human joint coordinate sequence at different scales, the processing results with feature scales of 81*1024, 27*1024, 9*1024, and 3*1024 are output respectively. Step 2: Using residual connections, the output of the dimensionality-upgraded results at different scales are respectively fused into each layer of a four-layer progressive network structure composed of multiple layers of depth-separable dilated residual convolutional modules connected in series, forming multi-stage supervision. The model depth is enhanced by constructing a multi-layer progressive layer structure through multiple layers of depth-separable dilated residual convolutional modules connected in series. In the depthwise separable dilated residual convolution module, the depthwise separable dilated residual convolution is first performed. This convolution consists of a point convolution module composed of a one-dimensional convolution with a kernel of 1, batch normalization, and the Mish activation function, and a convolution kernel of... And the expansion factor is The dilated channel-wise convolution module, consisting of channel-wise convolution, batch normalization, Mish activation function, and random deactivation, is executed. A slicing function is used to perform residual connection between the point convolution module and the dilated channel-wise convolution module to achieve depth-separable dilated residual convolution. The slicing function obtains information features that match the dimensions of the convolution before and after, which avoids overfitting and prevents network degradation. Then, a point convolution module and a random deactivation function are executed to fuse the information of each channel of the convolution and output the result. Step 3: In order to further explore the feature value of the shallow network, the shallow features output by the first layer up-dimensionality convolution module and the first layer depthwise separable dilated residual convolution module are respectively output to the depthwise separable strided residual convolution module for dimensionality reduction, and the shallow features are further optimized and refined to output deep features. The output of the first-layer dimensionality-upgrading convolutional module, after being dimensionality-reduced by the depthwise separable strided residual convolutional module, is input into the third and fourth-layer depthwise separable dilated residual convolutional modules, respectively. The output of the first-layer depthwise separable dilated residual convolutional module, after being dimensionality-reduced by the depthwise separable strided residual convolutional module, is then input into the fourth-layer depthwise separable dilated residual convolutional module. The shallow feature output and the deep feature input are connected by residual connecting blocks to achieve fine-grained supervision. The difference between this depthwise separable strided residual convolution module and the depthwise separable dilated residual convolution module lies in the dilation factor setting and stride setting; the depthwise separable strided residual convolution module does not use dilated convolution, and the stride is set to 3 and the dilation factor is 1. Step 4: The output of the fourth layer depth-separable dilated residual convolution module is fed to the fully connected layer. All feature information accumulated by the network is fused through the convolution of the fully connected layer, and the number of channels is reduced from 1024 to 51. Finally, a 1*3*17 three-dimensional human pose estimation result is output.

2. The method for 3D human pose estimation using a lightweight convolutional network according to claim 1, characterized in that, The process of upscaling the input two-dimensional joint coordinate sequence of the human body at different scales is specifically expressed as follows: in, Represents a two-dimensional human joint sequence. This indicates a higher-dimensional convolution module. This represents the slice function module, where k represents the k-th dimensionality increase process.

3. The method for 3D human pose estimation using a lightweight convolutional network according to claim 1, characterized in that, The depth-separable void residual module extracts the posture representation as follows: in, Represents a two-dimensional human pose sequence. represents a depthwise separable dilated residual convolutional module, and k represents the k-th depthwise separable dilated residual convolutional module.

4. The method for 3D human pose estimation using a lightweight convolutional network according to claim 1, characterized in that, The convolutional fusion in step 4 is represented as follows: in, This indicates a higher-dimensional convolution module. This represents a depth-separable dilated residual convolution module. This represents a depthwise separable strided residual convolution module.

5. A computer device comprising a processor and a memory, wherein the memory stores a computer program, characterized in that: When executed by a processor, the computer program implements a method for three-dimensional human pose estimation using a lightweight convolutional network as described in any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When a computer program is executed by a processor, it implements a method for three-dimensional human pose estimation using a lightweight convolutional network as described in any one of claims 1-4.

Images