Three-dimensional human pose estimation method and device based on depth separable dilated temporal convolution

Abstract

The application discloses a three-dimensional human body posture estimation method and device based on a deep separable dilated temporal convolution, constructs a T-layer deep separable dilated temporal convolution model, controls the receptive field of the model by setting a dilated rate size; in the two convolution layers of the T-layer deep separable dilated temporal convolution block, a slice function is used to match feature dimension information between the two convolution layers for residual connection, features of data in a matrix are extracted, and original features of the data are accumulated into the output of the deep separable dilated temporal convolution block, so that the features of the original data are reserved; the improved deep separable dilated convolution layer first performs point convolution, then performs channel-by-channel convolution, and a Mish activation function is introduced to improve the model performance, and the smooth and non-monotonic characteristics are increased; while reducing the model calculation complexity and the calculation amount, the model performance and the accuracy are improved, and the model calculation complexity and the model calculation amount are greatly reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and image video processing, and relates to a three-dimensional human pose estimation method and device based on deep separable dilated temporal convolution. BACKGROUND

[0002] The human pose estimation problem has always been a research hotspot in the field of computer vision, and the human pose estimation is a process of extracting human key point position information from a given image or a video. The general human pose estimation task can be divided into two categories: two-dimensional human pose estimation and three-dimensional human pose estimation.

[0003] In recent years, due to the successful application of deep learning in the field of human pose estimation, the model performance and generalization ability of two-dimensional human pose estimation have been significantly improved. Due to the fact that existing data sets are generally established indoors, the generalization performance of the three-dimensional human pose estimation model is limited due to the lack of large outdoor data sets. In the process of three-dimensional human pose estimation, the information is prone to be lost due to the influence of the ill-posed problem of two-dimensional to three-dimensional mapping.

[0004] In the process of human motion, the pose and appearance in the motion scene are full of variables, and human motion has a high degree of nonlinearity. The three-dimensional human pose estimation based on multi-camera and depth camera can effectively avoid the human joint occlusion problem, but the universality of the depth camera and the multi-camera is relatively low, and it has certain requirements for the shooting requirements and the scene and range. In daily life, the devices with photographing function basically have monocular camera function, so the applicable scene of the three-dimensional human pose estimation based on monocular camera is more extensive.

[0005] However, in the monocular video scene, there are the following problems: (1) the three-dimensional human pose precision is not high; (2) part of the information is lost in the two-dimensional to three-dimensional mapping process, resulting in a not smooth and coherent pose; (3) the network parameter amount is large, and other problems cause the three-dimensional human pose estimation method to easily appear the problem of incoherent pose in the monocular video scene, and cannot well utilize the information of the time dimension of the video. SUMMARY

[0006] In order to solve the above problems, the purpose of the present application is to provide a three-dimensional human pose estimation method and device based on deep separable dilated temporal convolution, which can realize the mapping of two-dimensional human pose to three-dimensional human pose, not only can reduce the use of network parameter amount, and can obtain high-precision pose estimation.

[0007] The three-dimensional human pose estimation method based on deep separable dilated temporal convolution of the present application comprises the following steps:

[0008] Step 1, for the input monocular human motion video, a two-dimensional pose detector is used to extract two-dimensional human joint coordinates, and a two-dimensional human joint coordinate prediction sequence is output;

[0009] Step 2, the input two-dimensional human joint coordinate prediction sequence is processed to increase the dimension, and the input J*2 key point data is increased to 1024, J represents the total number of predicted human joint points;

[0010] Step 3, a T-layer progressive depth separable hollow temporal convolution block is used to construct a T-layer depth separable hollow temporal convolution model, and the T-layer depth separable hollow temporal convolution block is executed, and the receptive field of the model is controlled by setting the hole rate size; in the two convolution layers of the T-layer depth separable hollow temporal convolution block, the slice function is used to match the feature dimension information between the two convolution layers for residual connection, the data in the matrix is extracted for feature, and the original feature of the data is accumulated into the output of the depth separable hollow temporal convolution block, so as to retain the features of the original data;

[0011] Each of the above depth separable hollow temporal convolution blocks is composed of two convolution layers, which are an improved depth separable hollow convolution layer and a one-dimensional conventional convolution layer, and the two convolution layers are connected by residual connection; first, the improved depth separable hollow convolution layer is executed, then the one-dimensional convolution layer is executed, and batch normalization, Mish activation function and random inactivation are performed after the one-dimensional convolution; after executing the T depth separable hollow temporal convolution blocks, a one-dimensional convolution is performed again, and the dimension is reduced to obtain 1*(3*J) output data, which is three-dimensional coordinate data representing J human joint points;

[0012] The improved depth separable hollow convolution layer includes point-by-point convolution, batch normalization, activation function, channel-by-channel convolution, batch normalization, activation function and random inactivation in sequence; in the improved depth separable hollow convolution layer, point-by-point convolution is first performed for channel information fusion, then channel-by-channel convolution is performed to process each channel information, and batch normalization and activation function operations are performed after point-by-point convolution and channel-by-channel convolution;

[0013] After obtaining the two-dimensional human joint coordinate prediction sequence of step 2, a dimension lifting convolution is first performed, and then T depth separable hollow temporal convolution blocks are performed, in each of which an improved depth separable hollow convolution layer is first performed, and then a one-dimensional convolution layer is performed, and after performing the T depth separable hollow temporal convolution blocks, a dimension reduction convolution module is entered to perform a dimension reduction convolution again, and the dimension reduction obtains 1*3*J output data, which is three-dimensional coordinate data representing J joint nodes of the human body; in the depth separable hollow temporal convolution block, on the one hand, a slicing function is used for dimension reduction processing, and the original feature of the data is accumulated to the output of the depth separable hollow temporal convolution time sequence module, and the original data is accumulated to form a residual connection, which avoids overfitting while retaining the original feature; on the other hand, in the residual connection of the T-1 depth separable hollow temporal convolution blocks, a slicing function and a dimension reduction convolution with a convolution kernel size of 1 and an output channel number of 3*J are used to further refine and optimize the predicted posture, and the result is accumulated to the output of the depth separable hollow temporal convolution time sequence model;

[0014] Step 4, in three-dimensional human posture estimation, an intermediate supervision strategy is adopted, the intermediate variable in the network is taken as part of the loss function, and the features at all levels of the network are further optimized, and the features at all levels are accumulated and fused to the output;

[0015] Step 5, the three-dimensional human joint coordinate sequence corresponding to the input monocular human motion video is output by the depth separable hollow convolution time sequence model, and constitutes a visual three-dimensional human posture estimation result.

[0016] The step 2 is specifically:

[0017] After obtaining the two-dimensional coordinate prediction sequence of J joints of each frame in the input layer, a dimension lifting convolution composed of a one-dimensional convolution, a batch regularization, an activation function and a random inactivation is sequentially performed, assuming that the input video sequence is 243 frames, that is, the input data is 243*17*2, a one-dimensional convolution with a convolution kernel size of 3, a hollow rate size of 3 and an output channel number of 1024 is performed on the input data, the output channel dimension is expanded to 1024, and then a batch regularization BatchNorm, an activation function Mish and a random inactivation Dropout are performed, and the output data is 241*1024.

[0018] The T=4.

[0019] The following three-dimensional human posture estimation loss function is constructed:

[0020]

[0021] wherein t represents the serial number of the current depth separable hollow temporal convolution block, T represents the total number of the depth separable hollow temporal convolution blocks, j represents the serial number of the current predicted joint, J represents the total number of the predicted human joints, w represents the weight of the current depth separable hollow temporal convolution block, and the loss function is used to calculate the loss of the three-dimensional human posture estimation.n weights representing the nth layer depth separable dilated temporal convolution module, three-dimensional human joint coordinate prediction value representing the model predicted frame, three-dimensional human joint coordinate ground truth value for the model predicted frame.

[0022] Further, the three-dimensional human pose estimation device based on the depth separable dilated temporal convolution comprises a video input module, a two-dimensional pose estimation module, a three-dimensional pose estimation module and a three-dimensional human pose estimation result output module connected in sequence; wherein

[0023] The video input module is used for inputting a monocular human motion video.

[0024] The two-dimensional pose estimation module is used for extracting two-dimensional human joint coordinates by using a two-dimensional pose detector on the input monocular human motion video, and outputting a two-dimensional human joint coordinate prediction sequence.

[0025] The three-dimensional pose estimation module comprises a dimension increasing convolution module, T progressive depth separable dilated temporal convolution blocks and a dimension decreasing convolution module connected in sequence, and the T depth separable dilated temporal convolution blocks are connected by using a residual connection, and a T layer depth separable dilated temporal convolution model is constructed by using the T depth separable dilated temporal convolution blocks.

[0026] Each depth separable dilated temporal convolution block comprises two convolution layers, i.e. an improved depth separable dilated convolution layer and a one-dimensional convolution layer, and the two convolution layers are connected by using a residual connection.

[0027] The improved depth separable dilated convolution layer comprises point-wise convolution, batch normalization, an activation function, channel-wise convolution, batch normalization, an activation function and random deactivation performed in sequence, in the improved depth separable dilated convolution layer, the point-wise convolution is performed first to fuse channel information, then the channel-wise convolution is performed to process each channel information, and the batch normalization and the activation function are performed after the point-wise convolution and the channel-wise convolution.

[0028] After obtaining the two-dimensional human joint coordinate prediction sequence, a dimension lifting convolution module is executed first, and then T depth separable hollow time sequence convolution blocks are executed, in the depth separable hollow time sequence convolution block, an improved depth separable hollow convolution layer is executed first, then a one-dimensional convolution layer is executed, and batch normalization, a Mish activation function and random inactivation are executed after the one-dimensional convolution; after executing the T depth separable hollow time sequence convolution blocks, a one-dimensional convolution is executed again, and the dimension is reduced to obtain 1*(3*J) output data, which is three-dimensional coordinate data representing J joint nodes of the human body; in the depth separable hollow time sequence convolution block, on the one hand, a slicing function is used for dimension reduction processing, and the original feature data is accumulated to the output of the depth separable hollow time sequence convolution module, and the residual connection is formed by accumulating the original data; on the other hand, in the residual connection of the T-1 depth separable hollow time sequence convolution blocks, a slicing function and a dimension reduction convolution with a convolution kernel size of 1 and an output channel number of 3*J are used to further refine and optimize the predicted posture, and the result is accumulated to the output of the depth separable hollow time sequence convolution model;

[0029] An intermediate supervision strategy is adopted, and the intermediate variable of the network is taken as part of the loss function to further optimize the features at each level of the network, and the features at each level are accumulated and fused to the output;

[0030] A three-dimensional human posture estimation result output module: outputting a three-dimensional human joint coordinate sequence corresponding to the input monocular human motion video, to form a visual three-dimensional human posture estimation result.

[0031] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the three-dimensional human posture estimation method based on the depth separable hollow time sequence convolution in any one of claims 1 to 4 when executing the computer program.

[0032] A computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the three-dimensional human posture estimation method based on the depth separable hollow time sequence convolution in any one of claims 1 to 4.

[0033] Compared with the prior art, the present application has the following advantages:

[0034] (1) The improved depth separable convolution is used in the model, which can greatly reduce the use of network parameter quantity and the calculation complexity;

[0035] (2) Based on the depth separable hollow time sequence convolution, the perception ability of the model can be improved, and the time dimension information is fully utilized, so that the predicted human posture is more smooth and coherent;

[0036] (3) The intermediate supervision scheme is adopted, the feature information of the shallow network is effectively utilized, the feature information of each level of the network is fused, and the performance of the model and the accuracy of the pose estimation are further improved.

[0037] The depth separable dilated convolution is adopted as the backbone network, the calculation complexity and the model calculation amount of the model are greatly reduced. Compared with the three-dimensional human pose estimation model based on the dilated convolution, the parameter amount is reduced by 50%, and the calculation complexity is only 76%. Meanwhile, in order to improve the performance of the model, the execution order of the depth separable dilated convolution is modified. The point convolution module is executed first, then the channel-by-channel convolution module is executed, and the Mish activation function is introduced to improve the performance of the model. Compared with the Relu function, the Mish function increases the smooth and non-monotonic characteristics, the smooth activation function can make the information better penetrate into the neural network, so that better accuracy and generalization are obtained. Therefore, while realizing the reduction of the calculation complexity and the calculation amount of the model, the performance and the accuracy of the model are improved. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 FIG. 1 is a schematic diagram of a three-dimensional human pose estimation network architecture based on a depth separable dilated convolutional kernel according to the present application;

[0039] Figure 2 FIG. 2 is a schematic diagram of an improved depth separable dilated convolution in a three-dimensional human pose estimation based on a depth separable dilated convolutional kernel according to the present application;

[0040] Figure 3 FIG. 3 is a schematic diagram of a depth separable dilated convolutional kernel block in a three-dimensional human pose estimation based on a depth separable dilated convolutional kernel according to the present application;

[0041] Figure 4 FIG. 4 is a system interface for online conversion of three-dimensional human poses of a video developed by the model according to the present application;

[0042] Figure 5 FIG. 5 is a flowchart of the present application.

[0043] The technical solutions in the embodiments of the present application will be further clearly and completely described below with reference to the drawings. DETAILED DESCRIPTION

[0044] As shown in FIG. 1, the present application provides a method for three-dimensional human pose estimation based on a depth separable dilated convolutional kernel, which comprises the following steps: Figures 1-5

[0045] Step 1, for the input monocular human motion video, a two-dimensional pose detector is used to extract two-dimensional human joint coordinates, and a two-dimensional human joint coordinate prediction sequence is output;

[0046] ​The two-dimensional pose detector adopting the cascade pyramid structure extracts a two-dimensional joint coordinate prediction sequence in the embodiment, and in the application, any specific two-dimensional key point detector is not relied on;

[0047] Step 2, dimensionality processing is performed on the input two-dimensional human joint coordinate prediction sequence, and the input 17*2 key point data is dimensionality processed to 1024;

[0048] After the two-dimensional human joint coordinate prediction sequence of J joints of each frame is obtained in the input layer, dimensionality processing is first performed by the dimensionality processing convolution composed of a one-dimensional convolution with a convolution kernel size of W and an output channel of C, batch normalization, an activation function and random inactivation. Assuming that the input video sequence is 243 frames, that is, the input data is 243*17*2, the one-dimensional convolution with a convolution kernel size of 3, a dilation rate size of 3 and an output channel number of 1024 is performed on the input data, the output channel dimension is expanded to 1024, and then the batch normalization BatchNorm, the activation function Mish and the random inactivation Dropout are performed, and the output data is 241*1024;

[0049] Step 3, based on the improved deep separable dilated convolution, a T=4 layer progressive convolution block is adopted to construct a T layer deep separable dilated temporal convolution model, the network depth is increased by using a plurality of layer progressive convolution blocks, so as to improve the ability of the model to extract high-dimensional feature information; T deep separable dilated temporal convolution blocks are executed, and the receptive field of the model is controlled by setting the dilation rate size; in the two convolution layers of the T layer deep separable dilated temporal convolution block, the slice function is used to match the feature dimension information between the two convolution layers for residual connection, the data in the matrix is extracted for feature, and the original features of the data are accumulated into the output of the deep separable dilated temporal convolution block, so as to retain the features of the original data;

[0050] In the embodiment, T=4 deep separable dilated temporal convolution blocks are adopted, each deep separable dilated temporal convolution block is composed of two convolution layers, which are an improved deep separable dilated convolution layer and a one-dimensional conventional convolution layer, and the two convolution layers are connected by using residual connection; the improved deep separable dilated convolution layer is executed first, then the one-dimensional convolution layer with a convolution kernel of 1 is executed, and the batch normalization, the Mish activation function and the random inactivation of 0.25 are executed after the one-dimensional convolution; after the T deep separable dilated temporal convolution blocks are executed, the one-dimensional convolution with a convolution kernel size of 1 and an output channel number of 17*3 (dimensionality reduction convolution) is executed again, and the dimensionality reduction obtains the output data of 1*51, which is the three-dimensional coordinate data representing 17 human joint nodes;

[0051] The improved deep separable dilated convolution layer comprises, in sequence, a point-by-point convolution, batch normalization, an activation function, a channel-by-channel convolution, batch normalization, an activation function and random inactivation, the convolution kernel of the point-by-point convolution is 1, the convolution kernel size of the channel-by-channel convolution is W, the number of channels is C and the dilated rate is D=W T .

[0052] The present application is different from the execution sequence of the conventional deep separable convolution, in the improved deep separable dilated convolution layer, the point-by-point convolution is first performed to fuse channel information, then the channel-by-channel convolution is performed to process each channel information, and batch normalization and an activation function are performed after the point-by-point convolution and the channel-by-channel convolution, and a smooth Mish activation function is introduced in the improved deep separable dilated convolution layer, which can make the feature information better penetrate into the network, thereby obtaining better accuracy and generalization.

[0053] In step 4, in three-dimensional human pose estimation, the low-level features of the network are also valuable, the present application adopts an intermediate supervision strategy, takes the intermediate variables of the network as part of the loss function, further optimizes the features at each level of the network, accumulates and fuses the features at each level to the output, realizes high-precision pose inference, and generates smoother three-dimensional human poses.

[0054] In the embodiment, the slice function is reused for residual connection of T-1 deep separable dilated convolution blocks to extract single-frame information, then a dimension reduction convolution with a convolution kernel of 1 and an output channel number of 3*17 is performed to further refine the feature information, and the refined feature information is accumulated to the output of the T-layer deep separable dilated convolution model, the loss function is calculated for each of the T deep separable dilated convolution blocks, the feature information at each level of the network is fused, the model realizes human pose inference from coarse to fine, and the following three-dimensional human pose estimation loss function is obtained by specific construction:

[0055]

[0056] wherein t represents the serial number of the current deep separable dilated convolution block, T represents the total number of the deep separable dilated convolution blocks, j represents the serial number of the current predicted joint, J represents the total number of the predicted joints, w n represents the weight of the n-layer deep separable dilated convolution module, represents the three-dimensional human joint coordinate prediction value predicted by the model for a frame, is the three-dimensional human joint coordinate true value predicted by the model for a frame.

[0057] The application is in the depth separable dilated temporal convolution temporal convolution block, on one hand, the slice function is adopted to reduce the dimension, the original feature of data is accumulated to the output of the depth separable dilated temporal convolution temporal module, and the residual connection is formed with the original data, the original feature is reserved while avoiding overfitting;On the other hand, in the residual connection of T-1 depth separable dilated temporal convolution blocks, the slice function and the dimension reduction convolution with the convolution kernel size of 1 and the output channel number of 3*17 are adopted to further refine and optimize the predicted posture, the result is accumulated to the output of the depth separable dilated temporal convolution model, and the more smooth three-dimensional human posture result is obtained through the fine prediction of the model.

[0058] Step 5, the three-dimensional human joint coordinate sequence corresponding to the output of the depth separable dilated convolution temporal model and the input monocular human motion video constitutes the visual three-dimensional human posture estimation result.

[0059] In the embodiment, as shown in Figure 4 The three-dimensional posture taking the pelvic bone as the root coordinate is output, the three-dimensional coordinates of the remaining joints are mapped to the three-dimensional space one by one and connected, and finally the visual three-dimensional human posture estimation result is output.

[0060] The three-dimensional human posture estimation device based on the depth separable dilated temporal convolution comprises a video input module, a two-dimensional posture estimation module, a three-dimensional posture estimation module and a three-dimensional human posture estimation result output module connected in sequence;Wherein

[0061] The video input module is used for inputting monocular human motion video;

[0062] The two-dimensional posture estimation module is used for extracting two-dimensional human joint coordinates by adopting a two-dimensional posture detector for the input monocular human motion video, and outputting a two-dimensional human joint coordinate prediction sequence;

[0063] The three-dimensional posture estimation module comprises a dimension increasing convolution module, T progressive depth separable dilated temporal convolution blocks and a dimension reduction convolution module connected in sequence, the T depth separable dilated temporal convolution blocks are connected through residual connection, and a T-layer depth separable dilated temporal convolution model is constructed through the T depth separable dilated temporal convolution blocks;

[0064] Each of the above deep separable dilated temporal convolution blocks is composed of two convolution layers, i.e., an improved deep separable dilated convolution layer and a one-dimensional conventional convolution layer, and the two convolution layers are connected by using a residual connection; the improved deep separable dilated convolution layer is executed first, then the one-dimensional convolution layer is executed, and batch normalization, a Mish activation function and random deactivation are executed after the one-dimensional convolution; after the T deep separable dilated temporal convolution blocks are executed, one-dimensional convolution is performed again, and dimension reduction is performed to obtain 1*(3*J) output data, which is three-dimensional coordinate data representing J joint nodes of a human body; in the deep separable dilated temporal convolution block, on the one hand, a slice function is used for dimension reduction processing, and original data features are accumulated to the output of the deep separable dilated temporal convolution time sequence module, and the original data are accumulated to form a residual connection, so that the original features are retained while overfitting is avoided; on the other hand, in the residual connection of the T-1 deep separable dilated temporal convolution blocks, a slice function and a dimension reduction convolution with a convolution kernel size of 1 and an output channel number of 3*J are used to further refine and optimize the predicted posture, and the result is accumulated to the output of the deep separable dilated temporal convolution time sequence model;

[0065] The improved deep separable dilated convolution layer includes, in sequence, point-by-point convolution, batch normalization, an activation function, channel-by-channel convolution, batch normalization, an activation function and random deactivation, the point-by-point convolution has a convolution kernel of 1, the channel-by-channel convolution has a convolution kernel size of W, a channel number of C and a dilated rate of D=W T In the improved deep separable dilated convolution layer, point-by-point convolution is first executed for channel information fusion, and then channel-by-channel convolution is executed to process each channel information, and batch normalization and an activation function are executed after the point-by-point convolution and the channel-by-channel convolution;

[0066] After the two-dimensional human joint coordinate prediction sequence is obtained, a dimension increasing convolution module with a convolution kernel size of W and an output channel of C is executed first, and then T deep separable dilated temporal convolution blocks are executed, each of which executes a dilated rate of D=W T The improved deep separable dilated convolution layer, a one-dimensional convolution layer with a convolution kernel of 1 is executed again, and batch normalization, an activation function and random deactivation are executed after the convolution, the size of the receptive field of the convolution layer is controlled by setting the dilated rate, and the depth of the model network is deepened by using the one-dimensional convolution layer; after the T deep separable dilated temporal convolution blocks are executed, a dimension reduction convolution module is entered to perform dimension reduction convolution, and dimension reduction is performed to obtain 1*51 output data, which is three-dimensional coordinate data representing 17 joint nodes of a human body; an intermediate supervision strategy is used, the intermediate variable in the network is taken as part of the loss function, the features at all levels of the network are further optimized, and the features at all levels are accumulated and fused to the output;

[0067] The three-dimensional human posture estimation result output module outputs a sequence of three-dimensional human joint coordinates corresponding to the input monocular human motion video, and forms a visual three-dimensional human posture estimation result.

[0068] The modules in the three-dimensional human posture estimation device based on deep separable dilated temporal convolution can be implemented by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so that the processor can call and execute the operations of the modules.

[0069] The three-dimensional human posture estimation device based on deep separable dilated temporal convolution can be used to perform the three-dimensional human posture estimation method based on deep separable dilated temporal convolution provided by any of the embodiments, and has the corresponding functions and advantages.

[0070] In one embodiment, a computer device, which can be a terminal, is provided. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a three-dimensional human posture estimation method based on deep separable dilated temporal convolution. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad provided on the housing of the computer device. The input device can also be an external keyboard, touchpad, or mouse, etc.

[0071] In one embodiment, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the three-dimensional human posture estimation method based on deep separable dilated temporal convolution provided by any of the embodiments, and has the corresponding functions and advantages.

[0072] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. The computer program is executed by a processor to implement the three-dimensional human posture estimation method based on deep separable dilated temporal convolution provided by any of the embodiments, and has the corresponding functions and advantages.

[0073] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0074] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0075] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method of 3D human pose estimation based on depth-wise separable dilated temporal convolution, characterized in that The method comprises the following steps: Step 1: for the input monocular human motion video, a two-dimensional pose detector is used to extract two-dimensional human joint coordinates, and a two-dimensional human joint coordinate prediction sequence is output; Step 2: the input two-dimensional human joint coordinate prediction sequence is processed to increase the dimension, and J*2 key point data is increased to 1024, and J represents the total number of predicted human joint nodes; Step 3: a T-layer progressive depth separable hollow temporal convolution block is used to construct a T-layer depth separable hollow temporal convolution model, and the T-layer depth separable hollow temporal convolution block is executed, and the receptive field of the model is controlled by setting the hollow rate; in the two convolution layers of the T-layer depth separable hollow temporal convolution block, a slice function is used to match the feature dimension information between the two convolution layers for residual connection, the data in the matrix is extracted for features, and the original features of the data are accumulated in the output of the depth separable hollow temporal convolution block, so that the features of the original data are retained; Each depth separable hollow temporal convolution block comprises two convolution layers, namely an improved depth separable hollow convolution layer and a one-dimensional conventional convolution layer, and the two convolution layers are connected by residual connection; the improved depth separable hollow convolution layer is executed first, then the one-dimensional convolution layer is executed, and batch normalization, Mish activation function and random inactivation are executed after the one-dimensional convolution; after the T depth separable hollow temporal convolution blocks are executed, one-dimensional convolution is performed again, and the dimension is reduced to obtain 1*(3*J) output data, which is three-dimensional coordinate data representing J human joint nodes; The improved depth separable hollow convolution layer comprises point-by-point convolution, batch normalization, activation function, channel-by-channel convolution, batch normalization, activation function and random inactivation performed in sequence; in the improved depth separable hollow convolution layer, point-by-point convolution is performed first to fuse channel information, then channel-by-channel convolution is performed to process each channel information, and batch normalization and activation function are performed after point-by-point convolution and channel-by-channel convolution; After obtaining the two-dimensional human joint coordinate prediction sequence of step 2, the dimension is first increased, then the T depth separable hollow temporal convolution blocks are executed, in each depth separable hollow temporal convolution block, the improved depth separable hollow convolution layer is executed first, then the one-dimensional convolution layer is executed, after the T depth separable hollow temporal convolution blocks are executed, the dimension is reduced again in the dimension reduction convolution module, and the dimension is reduced to obtain 1*3*J output data, which is three-dimensional coordinate data representing J human joint nodes; in the depth separable hollow temporal convolution block, on the one hand, the slice function is used for dimension reduction processing, and the original features of the data are accumulated in the output of the depth separable hollow temporal convolution block, and the residual connection is formed with the original data to avoid overfitting while retaining the original features; on the other hand, in the residual connection of the T-1 depth separable hollow temporal convolution blocks, the slice function and the dimension reduction convolution with a convolution kernel size of 1 and an output channel number of 3*J are used to further refine and optimize the predicted pose, and the result is accumulated in the output of the depth separable hollow temporal convolution model; Step 4: In the three-dimensional human pose estimation, an intermediate supervision strategy is adopted, the intermediate variable of the network is taken as part of the loss function, and the features at each level of the network are further optimized, and the features at each level are accumulated and fused to the output; Step 5: The three-dimensional human joint coordinate sequence corresponding to the input monocular human motion video is output by the deep separable dilated convolution time sequence model, and the three-dimensional human pose estimation result is constructed.

2. The method of claim 1, wherein the method is based on a deep separable dilated temporal convolution. The step 2 is specifically: After obtaining the two-dimensional coordinate prediction sequence of J joints of each frame in the input layer, the dimension increasing convolution composed of one-dimensional convolution, batch normalization, activation function and random inactivation is sequentially executed, assuming that the input video sequence is 243 frames, that is, the input data is 243*17*2, one-dimensional convolution with a convolution kernel size of 3, a dilated rate of 3 and an output channel number of 1024 is performed on the input data, the output channel dimension is expanded to 1024, and then batch normalization BatchNorm, activation function Mish and random inactivation Dropout are executed, and the output data is 241*1024.

3. The method of claim 1, wherein the method further comprises: determining a plurality of bounding boxes for the input image; and determining a plurality of bounding boxes for the output image. T=4。 4. The method of claim 1, wherein the method further comprises: The three-dimensional human pose estimation loss function is constructed as follows: wherein t represents the serial number of the current depth separable dilated temporal convolution block, T represents the total number of the depth separable dilated temporal convolution blocks, and j represents the serial number of the current predicted joint, represents the total number of predicted human body joints, represents the first layer depth separable dilated temporal convolution module weight, represents the three-dimensional human body joint coordinate prediction value of the model predicted frame, is the three-dimensional human body joint coordinate true value of the model predicted frame.

5. A device for 3D human pose estimation based on depth-wise separable dilated temporal convolution, characterized in that: It comprises a video input module, a two-dimensional pose estimation module, a three-dimensional pose estimation module and a three-dimensional human pose estimation result output module connected in sequence; The video input module is used for inputting a monocular human motion video; The two-dimensional pose estimation module is used for extracting two-dimensional human joint coordinates by using a two-dimensional pose detector on the input monocular human motion video, and outputting a two-dimensional human joint coordinate prediction sequence; The three-dimensional pose estimation module comprises a dimension increasing convolution module, T progressive deep separable dilated time convolution blocks and a dimension decreasing convolution module connected in sequence, and the T deep separable dilated time convolution blocks are connected by using a residual connection, and a T-layer deep separable dilated time convolution model is constructed by the T deep separable dilated time convolution blocks; Each deep separable dilated time convolution block comprises two convolution layers, that is, an improved deep separable dilated convolution layer and a one-dimensional convolution layer, and the two convolution layers are connected by using a residual connection; The improved deep separable dilated convolution layer comprises point-by-point convolution, batch normalization, activation function, channel-by-channel convolution, batch normalization, activation function and random inactivation performed in sequence, in the improved deep separable dilated convolution layer, point-by-point convolution is first performed for channel information fusion, then channel-by-channel convolution is performed to process each channel information, and batch normalization and activation function are performed after point-by-point convolution and channel-by-channel convolution; ​ ​ After obtaining the two-dimensional human joint coordinate prediction sequence, a dimension lifting convolution module is executed first, and then T depth separable hollow temporal convolution blocks are executed. In the depth separable hollow temporal convolution block, an improved depth separable hollow convolution layer is executed first, then a one-dimensional convolution layer is executed, and batch normalization, a Mish activation function and random deactivation are executed after the one-dimensional convolution. After executing the T depth separable hollow temporal convolution blocks, a one-dimensional convolution is executed again, and dimension reduction is performed to obtain 1*(3*J) output data, which is three-dimensional coordinate data representing J joint nodes of the human body. In the depth separable hollow temporal convolution block, on the one hand, a slicing function is used for dimension reduction processing, and the original feature data is accumulated to the output of the depth separable hollow temporal convolution block, and the original data is accumulated to form a residual connection; on the other hand, in the residual connection of the T-1 depth separable hollow temporal convolution blocks, a slicing function and a dimension reduction convolution with a convolution kernel size of 1 and an output channel number of 3*J are used to further refine and optimize the predicted posture, and the result is accumulated to the output of the depth separable hollow temporal convolution model; An intermediate supervision strategy is used, and intermediate variables in the network are used as part of the loss function to further optimize the features at each level of the network, and the features at each level are accumulated and fused to the output. The three-dimensional human pose estimation result output module outputs a three-dimensional human joint coordinate sequence corresponding to the input monocular human motion video, and constitutes a visual three-dimensional human pose estimation result. 6.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-5 when the computer program is executed by the processor. The processor executes the computer program to realize the three-dimensional human pose estimation method based on the depth separable hollow temporal convolution in any one of claims 1 to 4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the three-dimensional human pose estimation method based on the depth separable hollow temporal convolution in any one of claims 1 to 4.

Images