Human body posture estimation method, device and storage medium based on multi-feature fusion

Through the human posture estimation method of multi-feature fusion, the prediction subnet and the estimation subnet are combined with the Transformer and ChebGConv module to capture time and space information, solving the problem of degradation of recognition accuracy caused by depth blur and occlusion in the prior art, and achieving more Accurate position information acquisition.

CN117523661BActive Publication Date: 2024-11-26SUN YAT SEN UNIVERSITY SHENZHEN +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311485832.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-07
Publication Date
2024-11-26
Estimated Expiration
2043-11-07

AI Technical Summary

Technical Problem

The existing 3D human posture estimation method has a poor effect when facing depth blur or obstruction of human body parts, resulting in a decrease in recognition accuracy and making it difficult to effectively capture the time information in the video stream.

Method used

Using multi-feature fusion human pose estimation method, the predictive subnet and estimation subnet respectively extract the pose prediction features and pose estimation features, and fuse them. Using the Transformer model and ChebGConv module and other technologies, capture time and space information to realize the pose Complementary features.

Benefits of technology

The recognition accuracy of human posture estimation is improved, especially in the case of depth blur or occlusion, which can compensate for the reduction in the accuracy of predicting posture characteristics, obtain more accurate posture information, and enhance the robustness of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117523661B_ABST
    Figure CN117523661B_ABST
Patent Text Reader

Abstract

The present invention discloses a multi-feature fused human posture estimation method, a computer device and a storage medium, including the steps of obtaining predicted posture features and estimated posture features output by processing a human posture time series respectively by a prediction subnetwork and an estimation subnetwork, fusing the predicted posture features with the estimated posture features, and obtaining human posture fusion information. The present invention obtains human posture fusion information by fusing the predicted posture features with the estimated posture features. Generally, the predicted posture features can achieve good recognition accuracy. In the case where there is deep blur or occlusion of human body parts in the image, the predicted posture features with reduced accuracy are compensated by the estimated posture features, that is, the estimated posture features can be used as a supplement to the predicted posture features, avoiding the situation where incomplete feature extraction is caused by occlusion problems, realizing the complementary advantages of multiple features, and obtaining more accurate posture information. The present invention is widely used in the field of image processing technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a multi-feature fusion human body posture estimation method, a computer device and a storage medium. Background Art

[0002] 3D human pose estimation is a classic computer vision task that has been applied in a very wide range of fields. The content of the 3D human pose estimation task is to identify the pose of the human body contained in the image. At present, one of the implementation methods of 3D human pose estimation is the method of lifting from 2D to 3D. Specifically, this method uses a 2D pose estimation network to obtain 2D human coordinate points, and then inputs the 2D human coordinate points into the 3D pose estimation network to obtain the 3D human pose. The method of lifting from 2D to 3D can be flexibly combined with a 2D pose detector. Benefiting from the excellent performance of the most advanced 2D pose detector, the method of lifting from 2D to 3D is usually better than other 3D human pose estimation implementation methods such as direct estimation. However, the pictures or videos taken of the human body usually have depth blur or occlusion of human body parts, which will affect the mapping effect from 2D coordinates to 3D coordinates, thereby making the effect of human pose estimation worse. Summary of the invention

[0003] In view of the technical problems of low efficiency and poor versatility in current human body part detection technology, the purpose of the present invention is to provide a multi-feature fusion human body posture estimation method, computer device and storage medium.

[0004] On the one hand, an embodiment of the present invention includes a multi-feature fusion human body posture estimation method, including:

[0005] Get human posture time series;

[0006] Inputting the human body posture time series into the prediction subnetwork and the estimation subnetwork for processing respectively; wherein the prediction subnetwork is used to perform posture prediction according to the input two-dimensional human body posture time series, and the estimation subnetwork is used to extract time information and spatial information from the input two-dimensional human body posture time series;

[0007] Obtaining predicted posture features output by the prediction subnetwork;

[0008] Obtaining estimated posture features output by the estimation subnetwork;

[0009] The predicted posture feature is fused with the estimated posture feature to obtain human body posture fusion information.

[0010] Furthermore, the obtaining of a human body posture time series includes:

[0011] Obtain the two-dimensional coordinates detected by the two-dimensional posture estimation detector;

[0012] The two-dimensional coordinates are processed into a time series of human body postures.

[0013] Furthermore, the prediction subnetwork is used to perform several cycles. In one cycle, the prediction subnetwork performs the following steps:

[0014] Obtain previously predicted posture features;

[0015] Using an encoder module and an attention module in sequence, the previously predicted posture features and the human posture time series are processed to obtain a predicted heat map and an offset;

[0016] After sampling the predicted heat map, combine it with the offset to obtain the predicted posture feature of the current cycle process;

[0017] Output the predicted posture features obtained in the last round of the cycle process.

[0018] Further, the obtaining of previously predicted posture features includes:

[0019] When the current cycle is not the first cycle, obtaining the predicted posture feature obtained in the previous cycle as the previous predicted posture feature of the current cycle;

[0020] When the current cycle process is the first cycle process, an initial value is obtained as the previously predicted posture feature of the current cycle process.

[0021] Furthermore, the estimation subnetwork is used to:

[0022] The human body posture time series is processed using a ChebGConv module, a spatial transformer and a temporal transformer in sequence to obtain the estimated posture feature.

[0023] Further, the method of sequentially using a ChebGConv module, a spatial transformer, and a temporal transformer to process the human body posture time series to obtain the estimated posture feature includes:

[0024] Acquire a plurality of ChebGConv modules, a plurality of spatial transformers and a temporal transformer; wherein each of the ChebGConv modules has a corresponding spatial transformer;

[0025] Decomposing the human body posture time series into multiple frames of data;

[0026] Input each frame of data into a corresponding ChebGConv module and the spatial transformer respectively;

[0027] Acquire spatial information in a frame of input data extracted by the spatial transformer;

[0028] Input each of the spatial information into the time converter;

[0029] Acquire time information between each of the spatial information extracted by the spatial transformer;

[0030] The estimated posture feature is determined according to the time information and the spatial information.

[0031] Further, the obtaining of multiple ChebGConv modules, multiple spatial transformers and one temporal transformer comprises:

[0032] Get the self-attention based Transformer model;

[0033] The MLP layer in the Transformer model is replaced with a LAM-Gconv layer to obtain the spatial transformer.

[0034] Furthermore, fusing the predicted posture feature with the estimated posture feature to obtain human body posture fusion information includes:

[0035] Determine the frames to which the predicted posture feature and the estimated posture feature respectively correspond;

[0036] When the predicted posture feature and the estimated posture feature correspond to the same frame or a similar frame, the predicted posture feature and the estimated posture feature are fused to obtain the human body posture fusion information; otherwise, the predicted posture feature and the estimated posture feature are discarded.

[0037] On the other hand, an embodiment of the present invention also includes a computer device, including a memory and a processor, the memory is used to store at least one program, and the processor is used to load at least one program to execute a multi-feature fusion human body posture estimation method in the embodiment.

[0038] On the other hand, an embodiment of the present invention also includes a storage medium, which stores a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to execute a multi-feature fusion human body posture estimation method in the embodiment.

[0039] The beneficial effects of the present invention are as follows: the multi-feature fusion human body posture estimation method in the embodiment obtains human body posture fusion information by fusing the predicted posture feature with the estimated posture feature; since the human body posture fusion information includes both the predicted posture feature and the estimated posture feature, the predicted posture feature can generally achieve good recognition accuracy. When there is depth blur or occlusion of the human body part in the image, the predicted posture feature with reduced accuracy can be compensated by the estimated posture feature, that is, the estimated posture feature can be used as a supplement to the predicted posture feature to avoid incomplete feature extraction due to occlusion problems, thereby realizing the complementary advantages of multiple features and obtaining more accurate posture information. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A step diagram of a method for estimating a human body posture by fusion of multiple features in an embodiment;

[0041] Figure 2 Schematic diagram of the principle of the multi-feature fusion human body posture estimation method in the embodiment;

[0042] Figure 3 This is a schematic diagram of the structure of the prediction subnetwork in the embodiment;

[0043] Figure 4 It is a schematic diagram of the structure of the estimation sub-network in the embodiment;

[0044] Figure 5 FIG. 4 is a schematic diagram of the structure of the space transformer in the embodiment. DETAILED DESCRIPTION

[0045] The initial posture estimation network only uses the structural information of the posture itself for posture estimation. Although it can also obtain 3D human posture well, it cannot capture the time information in the frame sequence of the video stream when performing posture solidification on data in the form of video streams. Even if the convolutional neural network (CNN) or LSTM network is used to process the frame sequence of the video stream, although the time information can be captured to a certain extent, due to the limitations of the network structure such as CNN, it cannot capture long enough length dependencies, and can only obtain event correlation by capturing close frames and using a recurrent architecture. These technologies have limited temporal connectivity in nature, and these networks are also limited to simple sequential relationships when capturing temporal information. Therefore, it is difficult for networks such as CNN and LSTM to capture deep temporal information.

[0046] In addition to the problem of using temporal information, the self-occlusion problem of complex human movements is also a difficulty in posture estimation. The Transformer network has a self-attention mechanism that can capture global information and is very suitable for extracting temporal information features. The use of Transformer can rely on its comprehensive connectivity and expression to learn stronger temporal representations across frames, which will have a better effect on the 2D to 3D mapping problem caused by depth blur and occlusion.

[0047] Based on the above principle, in this embodiment, a multi-feature fusion human body posture estimation method is designed. Figure 1 ,The human posture estimation method based on multi-feature fusion includes the following steps:

[0048] S1. Obtain human body posture time series;

[0049] S2. Input the human body posture time series into the prediction sub-network and the estimation sub-network for processing respectively;

[0050] S3. Obtain the predicted posture features output by the prediction subnetwork;

[0051] S4. Obtain the estimated posture features output by the estimation subnetwork;

[0052] S5. Fuse the predicted posture features with the estimated posture features to obtain human body posture fusion information.

[0053] In this embodiment, the principles of steps S1-S5 are as follows Figure 2 See Figure 2 , using the prediction subnetwork, estimation subnetwork and feature fusion module to perform steps S1-S5.

[0054] When executing step S1, that is, the step of obtaining the human body posture time series, the following steps may be specifically performed:

[0055] S101. Obtain the two-dimensional coordinates detected by the two-dimensional posture estimation detector;

[0056] S102. Process the two-dimensional coordinates into a human body posture time series.

[0057] In step S101, the image data in the form of pictures or videos can be processed by a two-dimensional posture estimation detector and converted into two-dimensional coordinates. In step S102, the two-dimensional coordinates can be sorted according to the frames in which they are located to obtain a human posture time series. Therefore, in this embodiment, the human posture time series is two-dimensional data.

[0058] In step S2, the two-dimensional human posture time series obtained by executing step S1 is input into the prediction subnetwork and the estimation subnetwork respectively, and features are extracted by the networks of these two branches respectively.

[0059] In this embodiment, the structure of the prediction subnetwork used is as follows: Figure 3 See Figure 3 , the prediction subnetwork includes an encoder module (Encoder) and an attention module (Attention). The prediction subnetwork is modeled in an autoregressive manner and performs several rounds of loop processes. Taking any of the loop processes as an example, the prediction subnetwork performs the following steps: obtain the previous predicted posture features (if it is the first round of loop process, the previous predicted posture features can be set as the initial value; if it is not the first round of loop process, the predicted posture features obtained in the previous round of loop process are used as the previous predicted posture features of this round of loop process), use the encoder module (Encoder) and the attention module (Attention) in turn to process the previous predicted posture features and the human posture time series, and obtain the predicted heat map and offset output by the attention module (Attention); sample the predicted heat map and combine it with the offset to obtain the predicted posture features of this round of loop process; output the predicted posture features obtained in the last round of loop process.

[0060] The predicted posture features output by the prediction subnetwork can represent the precise posture coordinates obtained by identifying the human body posture time series.

[0061] In this embodiment, the structure of the estimation subnetwork used is as follows: Figure 4 See Figure 4 The estimation subnetwork includes multiple ChebGConv modules, multiple spatial transformers (Spatial Transformer) and a temporal transformer (TemporalTransformer).

[0062] Reference Figure 4, the estimation sub-network uses Transformer as the basic skeleton and uses ChebGConv module to replace the encoder layer of the traditional Transformer. After receiving the human posture time series, the estimation sub-network can decompose the human posture time series into multiple frames such as the first frame, the second frame, and the third frame, and then input the joint point coordinates corresponding to each frame into the corresponding ChebGConv module, for example, input the first frame data into ChebGConv module 1, and input the second frame data into ChebGConv module 2; the ChebGConv module inputs the processed data into the corresponding spatial transformer (SpatialTransformer), for example, ChebGConv module 1 inputs its processed data into spatial transformer 1, and ChebGConv module 2 inputs its processed data into spatial transformer 2; each spatial transformer (SpatialTransformer) extracts the spatial information between the joint points in the corresponding frame of data.

[0063] Reference Figure 4 , the spatial information output by each spatial transformer is input into the temporal transformer, which extracts the temporal information between multiple frames. Finally, the temporal information and spatial information constitute the estimated posture features to be output by the estimation subnetwork.

[0064] In this embodiment, Figure 4 The structure of the spatial transformer in the estimation subnetwork is shown as Figure 5 See Figure 5 The structure of the Spatial Transformer is similar to that of the traditional Transformer. Specifically, the MLP layer in the traditional Transformer can be replaced with the LAM-Gconv layer to obtain the Spatial Transformer. Figure 4 The Spatial Transformer shown in the figure can estimate the subnetwork by utilizing the graph structure information between joints, thereby outputting estimated pose features that contain invisible relationships between joints.

[0065] In this embodiment, when executing step S5, that is, fusing the predicted posture feature with the estimated posture feature to obtain human body posture fusion information, the following steps may be specifically performed:

[0066] S501. Determine the frames corresponding to the predicted posture features and the estimated posture features;

[0067] S502. When the predicted posture feature and the estimated posture feature correspond to the same frame or similar frames, the predicted posture feature and the estimated posture feature are fused to obtain human body posture fusion information; otherwise, the predicted posture feature and the estimated posture feature are discarded.

[0068] In step S502, key pixel comparison can be performed on the frames corresponding to the predicted posture feature and the estimated posture feature. For example, when the pixel values ​​of key pixel points exceeding the first threshold are the same, the frames corresponding to the predicted posture feature and the estimated posture feature can be considered to be similar frames; when the pixel values ​​of key pixel points exceeding the second threshold are the same, the frames corresponding to the predicted posture feature and the estimated posture feature can be considered to be the same frame. When the predicted posture feature and the estimated posture feature correspond to the same frame or similar frames, then Figure 2 The feature fusion module in the method fuses the predicted posture features with the estimated posture features to obtain the fusion information of the human body posture; if the frames corresponding to the predicted posture features and the estimated posture features are neither the same frame nor similar frames, the predicted posture features and the estimated posture features are not fused.

[0069] In this embodiment, the predicted posture features output by the prediction subnetwork obtained by executing step S3 can represent the precise posture coordinates obtained by identifying the human posture time series; the estimated posture features output by the estimation subnetwork obtained by executing step S4 include information on the invisible relationship between joints; by executing step S5, the predicted posture features are fused with the estimated posture features to obtain human posture fusion information; since the human posture fusion information includes both predicted posture features and estimated posture features, the predicted posture features can generally achieve good recognition accuracy. When there is depth blur or occlusion in the human body part in the image, the estimated posture features can compensate for the reduced accuracy of the predicted posture features, that is, the estimated posture features can serve as a supplement to the predicted posture features to avoid incomplete feature extraction due to occlusion problems, thereby achieving complementary advantages of multiple features and obtaining more accurate posture information.

[0070] A computer program for executing the multi-feature fusion method for human body posture estimation in the present embodiment can be written and written into a computer device or a storage medium. When the computer program is read out and executed, the multi-feature fusion method for human body posture estimation in the present embodiment is executed, thereby achieving the same technical effect as the multi-feature fusion method for human body posture estimation in the present embodiment.

[0071] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to the other feature, or it may be indirectly fixed or connected to the other feature. In addition, the descriptions of up, down, left, right, etc. used in the present disclosure are only relative to the relative positional relationship of the components of the present disclosure in the accompanying drawings. The singular forms of "a", "" and "the" used in the present disclosure are also intended to include the plural forms, unless the context clearly indicates other meanings. In addition, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as those generally understood by those skilled in the art. The terms used in the specification of this embodiment are only for describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this embodiment includes any combination of one or more related listed items.

[0072] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided by the present embodiment is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

[0073] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed dedicated integrated circuit for this purpose.

[0074] In addition, the operations of the process described in this embodiment may be performed in any suitable order, unless otherwise indicated in this embodiment or otherwise clearly contradicted by the context. The process described in this embodiment (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. A computer program includes a plurality of instructions that may be executed by one or more processors.

[0075] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the above steps in conjunction with a microprocessor or other data processor, the invention of this embodiment includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0076] The computer program can be applied to input data to perform the functions of the present embodiment, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on the display.

[0077] The above are only preferred embodiments of the present invention. The present invention is not limited to the above embodiments. As long as the technical effects of the present invention are achieved by the same means, any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical solutions and / or implementation methods may have various modifications and changes.

Claims

1. A multi-feature fusion human posture estimation method, characterized in that: The multi-feature fusion human posture estimation method comprises: Get human posture time series; The human body posture time series is input into the prediction sub-network and the estimation sub-network for processing respectively; wherein the prediction sub-network is used to perform posture prediction according to the input two-dimensional human body posture time series, and the estimation sub-network is used to extract time information and spatial information from the input two-dimensional human body posture time series; Obtaining predicted posture features output by the prediction subnetwork; Obtaining estimated posture features output by the estimation subnetwork; Fusing the predicted posture feature with the estimated posture feature to obtain human body posture fusion information; The estimation subnetwork is used to: Acquire a plurality of ChebGConv modules, a plurality of spatial transformers and a temporal transformer; wherein each of the ChebGConv modules has a corresponding spatial transformer; Decomposing the human body posture time series into multiple frames of data; Input each frame of data into a corresponding ChebGConv module and the spatial transformer respectively; Acquire spatial information in a frame of input data extracted by the spatial transformer; Input each of the spatial information into the time converter; Acquire time information between each of the spatial information extracted by the spatial transformer; The estimated posture feature is determined according to the time information and the spatial information.

2. The method for estimating human posture by multi-feature fusion according to claim 1, characterized in that: The step of obtaining a human body posture time series comprises: Obtain the two-dimensional coordinates detected by the two-dimensional posture estimation detector; The two-dimensional coordinates are processed into a time series of human body postures.

3. The method for estimating human posture by multi-feature fusion according to claim 1, characterized in that: The prediction subnetwork is used to perform several rounds of loop processes. During one round of the loop process, the prediction subnetwork performs the following steps: Obtain previously predicted posture features; Using an encoder module and an attention module in sequence, the previously predicted posture features and the human posture time series are processed to obtain a predicted heat map and an offset; After sampling the predicted heat map, combine it with the offset to obtain the predicted posture feature of the current cycle process; Output the predicted posture features obtained in the last round of the cycle process.

4. The method for estimating human posture by multi-feature fusion according to claim 3, characterized in that: The obtaining of previously predicted posture features includes: When the current cycle is not the first cycle, obtaining the predicted posture feature obtained in the previous cycle as the previous predicted posture feature of the current cycle; When the current cycle process is the first cycle process, an initial value is obtained as the previously predicted posture feature of the current cycle process.

5. The method for estimating human posture by multi-feature fusion according to claim 1, characterized in that: The step of obtaining a plurality of ChebGConv modules, a plurality of spatial transformers and a temporal transformer comprises: Get the self-attention based Transformer model; The MLP layer in the Transformer model is replaced with a LAM-Gconv layer to obtain the spatial transformer.

6. The method for estimating human posture by multi-feature fusion according to any one of claims 1 to 5, characterized in that: The step of fusing the predicted posture feature with the estimated posture feature to obtain human body posture fusion information includes: Determine the frames to which the predicted posture feature and the estimated posture feature respectively correspond; When the predicted posture feature and the estimated posture feature correspond to the same frame or a similar frame, the predicted posture feature and the estimated posture feature are fused to obtain the human body posture fusion information; otherwise, the predicted posture feature and the estimated posture feature are discarded.

7. A computer device, characterized in that: It includes a memory and a processor, the memory is used to store at least one program, and the processor is used to load at least one program to execute the multi-feature fusion human body posture estimation method of any one of claims 1-6.

8. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to execute the multi-feature fusion human body posture estimation method of any one of claims 1-6 when executed by the processor.