A pose estimation method and system based on a human structure guided learning network

Abstract

This invention discloses a pose estimation method and system based on a human structure-guided learning network, relating to the field of human pose estimation technology. The method includes: acquiring a human pose image to be measured; inputting the human pose image to a human pose estimation model to obtain the human pose; the human pose estimation model is trained using a training set and a calibrated deep convolutional neural network; the training set includes multiple first images and corresponding label information; the label information consists of human joints in the first images; the human joints are used to construct the human pose; the calibrated deep convolutional neural network includes a stem network, a first backbone network, a second backbone network, a human structure-guided learning branch network, and a feature fusion and refinement network. This invention combines a human structure-guided learning method and a feature fusion method to construct a human pose estimation model, improving the accuracy of model prediction.

Description

Technical Field

[0001] This invention relates to the field of human pose estimation technology, and in particular to a pose estimation method and system based on a human structure-guided learning network. Background Technology

[0002] As a fundamental and challenging problem in the field of computer vision, 2D human pose estimation plays different roles in various computer vision tasks and is widely used in areas such as abnormal behavior detection, human-computer interaction, and medical rehabilitation assistance. The goal of human pose estimation is to classify and locate the joints of human body parts, such as wrists, ankles, and torsos, from a given image or video to arrive at the correct human pose.

[0003] Despite the numerous studies and significant progress made in pose estimation techniques, current methods, whether top-down or bottom-up, all employ static pose estimation models and treat the network model as a black box. This results in low model flexibility, a lack of effective integration of prior knowledge about human anatomy, inability to effectively estimate pose in occluded scenarios, and poor interpretability. Summary of the Invention

[0004] The purpose of this invention is to provide a pose estimation method and system based on human structure-guided learning networks. By combining human structure-guided learning methods and feature fusion methods, a human pose estimation model is constructed, thereby improving the accuracy of model prediction.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A pose estimation method based on a human structure-guided learning network, the pose estimation method comprising:

[0007] Acquire an image of the human body posture to be tested; the image of the human body posture to be tested includes multiple human joint points;

[0008] The human pose image to be measured is input into the human pose estimation model to obtain the human pose.

[0009] The human pose estimation model is trained using a training set and a calibrated deep convolutional neural network. The training set includes multiple first images and corresponding label information for each first image. The label information consists of human joints in the first images. The human joints are used to construct the human pose. The calibrated deep convolutional neural network includes a stem network, a first backbone network, a second backbone network, a human structure-guided learning branch network, and a feature fusion and refinement network.

[0010] The stem network's input is used to input the training set; the stem network's output is connected to the input of the first backbone network; the stem network is used to extract features from the first image in the training set to determine a first feature map.

[0011] The first output of the first backbone network is connected to the input of the second backbone network, and the second output of the first backbone network is connected to the input of the human structure guided learning branch network; the first backbone network is used to extract features from the first feature map to determine the second feature map.

[0012] The output of the second backbone network is connected to the first input of the feature fusion and refinement network. The second backbone network is used to extract features from the second feature map to determine the third feature map.

[0013] The output of the human structure guided learning branch network is connected to the second input of the feature fusion and refinement network; the human structure guided learning branch network is used to perform human structure analysis on the second feature map to determine the fourth feature map.

[0014] The feature fusion and refinement network is used to perform multi-scale feature fusion on the third feature map and the fourth feature map, and to refine the pose of the fused feature map to determine the human pose.

[0015] Optionally, the human structure-guided learning branch network includes an encoding subnetwork and a decoding subnetwork;

[0016] The input of the encoding subnetwork is connected to the second output of the first backbone network, the output of the encoding subnetwork is connected to the input of the decoding subnetwork, and the output of the decoding subnetwork is connected to the second input of the feature fusion and refinement network.

[0017] The encoding subnetwork includes multiple encoding modules connected in sequence, and the decoding subnetwork includes multiple decoding modules connected in sequence; the encoding module and the decoding module correspond one-to-one, and one encoding module is residually connected to one of the corresponding decoding modules;

[0018] The encoding module includes a first memory-enabled Transformer module and a downsampling module connected in sequence; the first memory-enabled Transformer module includes a non-overlapping window self-attention mechanism network and a forward propagation network with memory units;

[0019] The decoding module includes an upsampling module and a second memory-based Transformer module connected in sequence; the second memory-based Transformer module includes a non-overlapping window self-attention mechanism network and a forward propagation network with memory units.

[0020] Optionally, the feature fusion and refinement network includes a multi-scale fusion module and a pose refinement module;

[0021] The first input terminal of the multi-scale fusion module is connected to the output terminal of the second backbone network, the second input terminal of the multi-scale fusion module is connected to the output terminals of the multiple decoding modules respectively, and the output terminal of the multi-scale fusion module is connected to the input terminal of the pose refinement module.

[0022] The output of the posture refinement module is used to output the human posture.

[0023] The multi-scale fusion module includes a feature fusion submodule and multiple feature extraction submodules; the number of feature extraction submodules is the same as the number of decoding modules, and one feature extraction submodule corresponds to one decoding module;

[0024] The first input terminal of the feature extraction submodule is connected to the output terminal of the second backbone network; the second input terminal of the feature extraction submodule is connected to the output terminal of the corresponding decoding module; the output terminal of the feature extraction submodule is connected to the input terminal of the feature fusion submodule; and the output terminal of the feature fusion submodule is connected to the input terminal of the pose refinement module.

[0025] The feature fusion submodule is used to fuse the sub-feature maps output by multiple feature extraction submodules to determine the final feature map;

[0026] The posture refinement module is used to determine multiple human joints based on the final feature map, and to determine human posture based on the multiple human joints.

[0027] Optionally, the training process of the human pose estimation model further includes:

[0028] Determine the target loss function;

[0029] The calibrated deep convolutional neural network is trained according to the target loss function and the training set to obtain a human pose estimation model;

[0030] The target loss function is determined by a first loss function, a second loss function, a third loss function, and a fourth loss function. The first loss function represents the loss value between the predicted human joint point corresponding to the second feature map and the human joint point in the first image. The second loss function represents the loss value between the offset of the predicted human joint point corresponding to the second feature map and the average offset of the predicted human joint point. The third loss function represents the loss value between the distance between each predicted human joint point in the predicted human joint point corresponding to the fourth feature map and the distance between each predicted human joint point in the first image. The fourth loss function represents the loss value between the predicted human joint point corresponding to the fused feature map and the human joint point in the first image.

[0031] Optionally, the first loss function is:

[0032]

[0033] Among them, L h Let represent the first loss function value; i represents the human body number, j represents the human body joint number, and i and j are positive integers; P i (p j ) represents the predicted joint point of the i-th person in the second feature map; This represents the heatmap of the joint points corresponding to the second feature map. G represents the coarse joints in the joint heatmap corresponding to the second feature map, where λ represents the number of joints; i (p j ) represents the actual human joints of the i-th person in the first image; G = {g1,...,g λ} represents the heatmap of the joint points corresponding to the first image, g λ This represents the actual human joint points in the joint point heatmap corresponding to the first image;

[0034] The second loss function is:

[0035] Among them, L o Indicates the value of the second loss function; O i (p j ) represents the offset of the predicted human joint point of the i-th person in the second feature map;

[0036] The third loss function is:

[0037] Among them, L b The value of the third loss function is represented; k and n represent the numbers of the human joints in the fourth feature map, and k,n∈j; This represents the k-th human joint predicted in the fourth feature map. This represents the nth human joint point predicted in the fourth feature map; This represents the actual k-th human body joint in the first image. This represents the actual nth human body joint in the first image; This represents the actual distance between adjacent human body joints. This indicates the predicted distance between adjacent human body joints;

[0038] The fourth loss function is:

[0039] Among them, L m M represents the value of the fourth loss function. i (m j ) represents the predicted human joint point of the i-th person in the fused feature map; This represents the joint heatmap corresponding to the fused feature map. G represents the coarse joints in the joint heatmap corresponding to the fused feature map, where λ represents the number of joints; i (p j ) represents the actual human joints of the i-th person in the first image; G = {g1,...,g λ} represents the heatmap of the joint points corresponding to the first image, g λ This represents the actual human joint in the joint point heatmap corresponding to the first image.

[0040] Optionally, the process of determining the target loss function specifically includes:

[0041] According to the formula L=αL h +βL o +γL b +δL m Determine the target loss function;

[0042] Where α represents the weight of the first loss function value, β represents the weight of the second loss function value, γ represents the weight of the third loss function value, and δ represents the weight of the fourth loss function value.

[0043] Optionally, the first backbone network includes a first subnet and a second subnet;

[0044] The input of the first subnet is connected to the output of the stem network, and the output of the first subnet is connected to the input of the second subnet; the first output of the second subnet is connected to the input of the second backbone network, and the second output of the second subnet is connected to the input of the human structure guided learning branch network.

[0045] The first subnet includes multiple residual modules connected in sequence;

[0046] The second subnet includes a first branch and a second branch; both the first branch and the second branch include multiple residual modules, and the number of residual modules in the first branch is the same as the number of residual modules in the second branch.

[0047] The input of the first branch is connected to the first output of the first subnet, the first output of the first branch is connected to the input of the second backbone network, and the second output of the first branch is connected to the input of the human structure guided learning branch network.

[0048] The second branch is derived from the first subnet through downsampling; the input of the second branch is connected to the second output of the first subnet, the first output of the second branch is connected to the input of the second backbone network, and the second output of the second branch is connected to the input of the human structure guided learning branch network.

[0049] Optionally, the second backbone network includes a third subnet and a fourth subnet;

[0050] The first input terminal of the third subnet is connected to the first branch, and the second input terminal of the third subnet is connected to the second branch; the output terminal of the third subnet is connected to the input terminal of the fourth subnet, and the output terminal of the fourth subnet is connected to the first input terminal of the feature fusion and refinement network.

[0051] The third subnet includes parallel third, fourth, and fifth branches; each of the third, fourth, and fifth branches includes multiple residual modules, and the number of residual modules in the third branch, the fourth branch, and the fifth branch are all the same.

[0052] The input terminal of the third branch is connected to the first output terminal of the first branch and the first output terminal of the second branch, respectively; the input terminal of the fourth branch is connected to the first output terminal of the first branch and the first output terminal of the second branch, respectively.

[0053] The fifth branch is derived from the second branch through downsampling; the input of the fifth branch is connected to the first output of the first branch and the first output of the second branch, respectively.

[0054] The fourth subnet includes parallel sixth, seventh, eighth and ninth branches; each of the sixth, seventh, eighth and ninth branches includes multiple residual modules, and the number of residual modules in the sixth branch, the seventh branch, the eighth branch and the ninth branch are all the same.

[0055] The input terminal of the sixth branch is connected to the output terminals of the third branch, the fourth branch, and the fifth branch, respectively; the input terminal of the seventh branch is connected to the output terminals of the third branch, the fourth branch, and the fifth branch, respectively; the input terminal of the eighth branch is connected to the output terminals of the third branch, the fourth branch, and the fifth branch, respectively.

[0056] The ninth branch is derived from the fifth branch through downsampling; the input of the ninth branch is connected to the output of the third branch, the output of the fourth branch, and the output of the fifth branch, respectively.

[0057] To achieve the above objectives, the present invention also provides the following technical solutions:

[0058] A pose estimation system based on a human structure-guided learning network includes:

[0059] An image acquisition subsystem is used to acquire an image of the human body posture to be tested; the image of the human body posture to be tested includes multiple human joint points.

[0060] The pose estimation subsystem is used to input the human pose image to be measured into the human pose estimation model in order to obtain the human pose.

[0061] The human pose estimation model is trained using a training set and a calibrated deep convolutional neural network. The training set includes multiple first images and corresponding label information for each first image. The label information consists of human joints in the first images. The human joints are used to construct the human pose. The calibrated deep convolutional neural network includes a stem network, a first backbone network, a second backbone network, a human structure-guided learning branch network, and a feature fusion and refinement network.

[0062] The stem network's input is used to input the training set; the stem network's output is connected to the input of the first backbone network; the stem network is used to extract features from the first image in the training set to determine a first feature map.

[0063] The first output of the first backbone network is connected to the input of the second backbone network, and the second output of the first backbone network is connected to the input of the human structure guided learning branch network; the first backbone network is used to extract features from the first feature map to determine the second feature map.

[0064] The output of the second backbone network is connected to the first input of the feature fusion and refinement network. The second backbone network is used to extract features from the second feature map to determine the third feature map.

[0065] The output of the human structure guided learning branch network is connected to the second input of the feature fusion and refinement network; the human structure guided learning branch network is used to perform human structure analysis on the second feature map to determine the fourth feature map.

[0066] The feature fusion and refinement network is used to perform multi-scale feature fusion on the third feature map and the fourth feature map, and to refine the pose of the fused feature map to determine the human pose.

[0067] Optionally, the human structure-guided learning branch network includes an encoding subnetwork and a decoding subnetwork;

[0068] The input of the encoding subnetwork is connected to the second output of the first backbone network, the output of the encoding subnetwork is connected to the input of the decoding subnetwork, and the output of the decoding subnetwork is connected to the second input of the feature fusion and refinement network.

[0069] The encoding subnetwork includes multiple encoding modules connected in sequence, and the decoding subnetwork includes multiple decoding modules connected in sequence; the encoding module and the decoding module correspond one-to-one, and one encoding module is residually connected to one of the corresponding decoding modules;

[0070] The encoding module includes a first memory-enabled Transformer module and a downsampling module connected in sequence; the first memory-enabled Transformer module includes a non-overlapping window self-attention mechanism network and a forward propagation network with memory units;

[0071] The decoding module includes an upsampling module and a second memory-based Transformer module connected in sequence; the second memory-based Transformer module includes a non-overlapping window self-attention mechanism network and a forward propagation network with memory units.

[0072] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0073] This invention constructs a human pose estimation model by combining a stem network, a first backbone network, a second backbone network, a human structure-guided learning branch network, and a feature fusion and refinement network. The human structure-guided learning branch network is a dynamic structure network that can effectively learn and adapt to pose parameters in a single step, thereby quickly adapting to the basic pose estimation model to assist in the detection and estimation of human joints. The feature fusion and refinement network effectively fuses features from a fourth feature map with rich semantic information into a third feature map with strong spatial location information, thereby improving the accuracy of joint prediction. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 This is a flowchart illustrating the pose estimation method based on a human structure-guided learning network according to the present invention.

[0076] Figure 2 This is a schematic diagram of the two-scale feature fusion modules in the feature fusion and refinement network of the present invention;

[0077] Figure 3 This is a schematic diagram of a multi-scale progressive feature fusion module in the feature fusion refinement network of the present invention;

[0078] Figure 4 This is a schematic diagram of the pose estimation system based on a human structure-guided learning network according to the present invention.

[0079] Figure 5 This is a schematic diagram of the human posture estimation model of the present invention. Detailed Implementation

[0080] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0081] The purpose of this invention is to provide a pose estimation method and system based on a human structure-guided learning network. It adopts a Transformer-based Encoder-Decoder network structure as a human structure-guided learner, which effectively learns and adapts to pose parameters in one go, thereby quickly adapting to the basic pose estimation model to assist in the prediction of joints. Through a multi-scale feature fusion method based on an attention mechanism, it effectively fuses high-level features with rich semantic information into high-resolution feature maps with strong spatial location information, thereby improving the accuracy of joint prediction.

[0082] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0083] Example 1

[0084] like Figure 1 As shown, this invention provides a pose estimation method based on a human structure-guided learning network, comprising:

[0085] Step 100: Obtain the human posture image to be tested; the human posture image to be tested includes multiple human joints.

[0086] Step 200: Input the human pose image to be tested into the human pose estimation model to obtain the human pose.

[0087] The human pose estimation model is trained using a training set and a calibrated deep convolutional neural network. The training set includes multiple first images and corresponding label information. The label information consists of human joints in the first images. These human joints are used to construct the human pose. The calibrated deep convolutional neural network includes a stem network, a first backbone network, a second backbone network, a human structure-guided learning branch network, and a feature fusion and refinement network. Specifically, the stem network comprises two 3×3 convolutional kernels with stride values. The stem network is used to change the channels of the image feature map and the image resolution, thereby extracting image features.

[0088] The stem network's input is used to input the training set; the stem network's output is connected to the input of the first backbone network; the stem network is used to extract features from the first image in the training set to determine a first feature map. The first output of the first backbone network is connected to the input of the second backbone network, and the second output of the first backbone network is connected to the input of the human structure guided learning branch network; the first backbone network is used to extract features from the first feature map to determine a second feature map. The output of the second backbone network is connected to the first input of the feature fusion and refinement network, and the second backbone network is used to extract features from the second feature map to determine a third feature map.

[0089] The output of the human structure guided learning branch network is connected to the second input of the feature fusion and refinement network. The human structure guided learning branch network is used to perform human structure analysis on the second feature map to determine the fourth feature map. The feature fusion and refinement network is used to perform multi-scale feature fusion on the third and fourth feature maps, and to refine the pose of the fused feature map to determine the human pose.

[0090] Specifically, before training the human pose estimation model, sample images (the first image) and corresponding annotation data for each sample image are acquired from the dataset. Specifically, the sample images are processed by a human detection model to extract each individual image containing a single person, and the images are resized to 384x288 RGB images. Then, the keypoint positions annotated in the original dataset are adjusted to the scale of the input image, and a ground truth heatmap is constructed. Specifically, adjusting the keypoint positions annotated in the original dataset to the scale of the input image reduces potential errors during training.

[0091] During the training of the human pose estimation model, the training set is input into a calibrated deep convolutional neural network (DCNN) for training, and then the trained DCNN is validated using a validation set. During the training of the DCNN, a small initial accuracy is set, and the detection accuracy of the human detection model after each training iteration is compared with the current best accuracy. If the human detection model obtained in the current iteration has the highest accuracy, this model is saved, and its accuracy is taken as the current best accuracy. In the next iteration, the network parameters of the calibrated DCNN are updated according to the loss value of the target loss function, and the calibrated DCNN begins the next iteration of training. When the number of training iterations reaches the preset number of training iterations, training ends, and the model with the highest accuracy across all training iterations is taken as the final human detection model. In one specific embodiment, the preset number of training iterations is 240.

[0092] Preferably, such as Figure 5 As shown, the human structure-guided learning branch network includes an encoding subnetwork and a decoding subnetwork; the input of the encoding subnetwork is connected to the second output of the first backbone network, the output of the encoding subnetwork is connected to the input of the decoding subnetwork, and the output of the decoding subnetwork is connected to the second input of the feature fusion and refinement network.

[0093] The encoding subnetwork includes multiple encoding modules connected in sequence, and the decoding subnetwork includes multiple decoding modules connected in sequence; the encoding modules and the decoding modules correspond one-to-one, and in order to prevent gradient vanishing, one encoding module is residually connected to one of the corresponding decoding modules.

[0094] In one specific embodiment, the encoding subnetwork includes four encoding modules, and the decoding subnetwork includes four decoding modules. Each encoding module includes a first MemoryTransformer Block (MTB) and a downsampling module connected in sequence; the first MTB includes a non-overlapping window self-attention mechanism network and a forward propagation network with memory units. Each decoding module includes an upsampling module and a second MTB; the second MTB includes a non-overlapping window self-attention mechanism network and a forward propagation network with memory units.

[0095] Specifically, the encoding sub-network includes a first MTB, a first 1 / 2 downsampling module, a third MTB, a second 1 / 2 downsampling module, a fifth MTB, a third 1 / 2 downsampling module, a seventh MTB, and a fourth 1 / 2 downsampling module connected in sequence; the decoding sub-network includes a first ×2 upsampling module, a second MTB, a second ×2 upsampling module, a fourth MTB, a third ×2 upsampling module, a sixth MTB, a fourth ×2 upsampling module, and an eighth MTB connected in sequence. The first MTB is also skip-connected to the eighth MTB, performing feature addition to obtain a first output feature, which is then output to the feature fusion and refinement network; the first output feature is the human pose feature extracted via convolutional kernels. The third MTB is also connected to the sixth MTB in a skip connection, performing feature addition to obtain a second output feature, and outputting the second output feature to the feature fusion and refinement network; the fifth MTB is also connected to the fourth MTB in a skip connection, performing feature addition to obtain a third output feature, and outputting the third output feature to the feature fusion and refinement network; the seventh MTB is also connected to the second MTB in a skip connection, performing feature addition to obtain a fourth output feature, and outputting the fourth output feature to the feature fusion and refinement network.

[0096] The human structure guided learning branch network composed of the above four encoding modules and four decoding modules can achieve higher accuracy in human joint estimation; to achieve faster human joint estimation speed, a human structure guided learning branch network composed of three encoding modules and three decoding modules can be constructed.

[0097] Preferably, the first backbone network includes a first subnet and a second subnet. The first subnet has 48 channels; the second subnet is a parallel multi-resolution network with two branches. The input of the first subnet is connected to the output of the stem network, and the output of the first subnet is connected to the input of the second subnet. The first output of the second subnet is connected to the input of the second backbone network, and the second output of the second subnet is connected to the input of the human structure guided learning branch network.

[0098] The first subnet includes multiple residual modules connected in sequence; specifically, the first subnet includes four residual modules arranged horizontally in parallel and connected. The second subnet includes a first branch and a second branch; both the first branch and the second branch include multiple residual modules, and the number of residual modules in the first branch is the same as the number of residual modules in the second branch, which is four.

[0099] The input of the first branch is connected to the first output of the first subnet, the first output of the first branch is connected to the input of the second backbone network, and the second output of the first branch is connected to the input of the human structure guided learning branch network. Specifically, the second output of the first branch is connected to the input of the first MTB. The second branch is determined by downsampling from the first subnet; the input of the second branch is connected to the second output of the first subnet, the first output of the second branch is connected to the input of the second backbone network, and the second output of the second branch is connected to the input of the human structure guided learning branch network. Specifically, the second output of the second branch is connected to the input of the first MTB.

[0100] The second backbone network includes a third subnetwork and a fourth subnetwork; wherein the third subnetwork is a parallel multi-resolution network with 3 branches, and the fourth subnetwork is a parallel multi-resolution network with 4 branches. The first input terminal of the third subnetwork is connected to the first branch, and the second input terminal of the third subnetwork is connected to the second branch; the output terminal of the third subnetwork is connected to the input terminal of the fourth subnetwork, and the output terminal of the fourth subnetwork is connected to the first input terminal of the feature fusion and refinement network.

[0101] The third subnet includes parallel third, fourth, and fifth branches; each of the third, fourth, and fifth branches includes multiple residual modules, and the number of residual modules in the third branch, the fourth branch, and the fifth branch is the same, which is four. The input terminal of the third branch is connected to the first output terminal of the first branch and the first output terminal of the second branch, respectively; the input terminal of the fourth branch is connected to the first output terminal of the first branch and the first output terminal of the second branch, respectively.

[0102] The fifth branch is derived from the second branch through downsampling; the input of the fifth branch is connected to the first output of the first branch and the first output of the second branch, respectively.

[0103] The fourth subnet includes parallel sixth, seventh, eighth and ninth branches; each of the sixth, seventh, eighth and ninth branches includes multiple residual modules, and the number of residual modules in the sixth branch, the seventh branch, the eighth branch and the ninth branch are all the same, which is 4.

[0104] The input terminal of the sixth branch is connected to the output terminals of the third branch, the fourth branch, and the fifth branch, respectively; the input terminal of the seventh branch is connected to the output terminals of the third branch, the fourth branch, and the fifth branch, respectively; and the input terminal of the eighth branch is connected to the output terminals of the third branch, the fourth branch, and the fifth branch, respectively.

[0105] The ninth branch is derived from the fifth branch through downsampling; the input of the ninth branch is connected to the output of the third branch, the output of the fourth branch, and the output of the fifth branch, respectively.

[0106] Furthermore, the first subnet, the second subnet, the third subnet, and the fourth subnet are connected sequentially from left to right, and the branches in each subnet are arranged vertically and aligned.

[0107] Preferably, the feature fusion and refinement network includes a multi-scale fusion module and a pose refinement module; the first input of the multi-scale fusion module is connected to the output of the second backbone network, the second input of the multi-scale fusion module is connected to the outputs of multiple decoding modules respectively, and the output of the multi-scale fusion module is connected to the input of the pose refinement module; the output of the pose refinement module is used to output human pose.

[0108] The multi-scale fusion module includes a feature fusion submodule and multiple feature extraction submodules; the number of feature extraction submodules is the same as the number of decoding modules, and one feature extraction submodule corresponds to one decoding module, with four of them.

[0109] The first input terminal of the feature extraction submodule is connected to the output terminal of the second backbone network, the second input terminal of the feature extraction submodule is connected to the output terminal of the corresponding decoding module, the output terminal of the feature extraction submodule is connected to the input terminal of the feature fusion submodule, and the output terminal of the feature fusion submodule is connected to the input terminal of the pose refinement module.

[0110] The first feature extraction submodule has its first input connected to the output of the sixth branch. Its second input is used to input the first output convolutional kernel from the decoding module. The first feature extraction submodule applies the first output convolutional kernel to the feature map output by the sixth branch to extract first human structural feature information and output first scale features. The second feature extraction submodule has its first input connected to the output of the seventh branch. Its second input is used to input the second output convolutional kernel from the decoding module. The second feature extraction submodule applies the second output convolutional kernel to the feature map output by the seventh branch to extract second human structural feature information and output second scale features. The third feature extraction submodule has its first input connected to the output of the eighth branch. Its second input is used to input the third output convolutional kernel from the decoding module. The third feature extraction submodule applies the third output convolutional kernel to the feature map output by the eighth branch to extract third human structural feature information and output third scale features. The first input of the fourth feature extraction submodule is connected to the output of the ninth branch. The second input of the fourth feature extraction submodule is used to input the fourth output convolutional kernel from the decoding module. The fourth feature extraction submodule applies the fourth output convolutional kernel to the feature map output by the ninth branch to extract the fourth human structural feature information and output the fourth scale feature. The feature maps output by the sixth, seventh, eighth, and ninth branches are feature maps of different resolutions. The first, second, third, and fourth output convolutional kernels are applied to the feature maps output by each branch to extract specific human structural feature information.

[0111] The feature fusion submodule is used to fuse the sub-feature maps output by multiple feature extraction submodules to determine the final feature map; the pose refinement module is used to determine multiple human joints based on the final feature map, and to determine the human pose based on the multiple human joints.

[0112] Specifically, the first-scale features, second-scale features, third-scale features, and fourth-scale features are input into the feature fusion submodule. For example... Figure 2 and Figure 3 As shown, in the feature fusion submodule, the third-scale features are subjected to self-similarity calculation to obtain an output Sout with the same resolution as the third-scale features. 3,1 Features; similarity calculation is performed between the third-scale features and the fourth-scale features to obtain an output Sout with the same resolution as the third-scale features. 3,2 Features; including third-scale features, Sout 3,1 Features and Sout 3,2 The features are concatenated along the channel dimension and then passed through a 1x1 convolutional layer to output Sout. 3,f Features. Similarly, recursively calculate the second-scale features and Sout. 3,f Sout after feature fusion 2,f Similarly, recursively calculate the first-scale features and Sout. 2,f Sout after feature fusion 1,f Features; Sout 1,f The features are determined as the final feature map, thereby realizing the fusion of high-level feature maps with rich semantic information into low-level feature map pixels with strong spatial information.

[0113] Finally, the final feature map is input into the pose refinement module, which adjusts the feature map channels to the same number of joints, generates the final predicted joint heatmap, and determines the human pose.

[0114] Furthermore, to accelerate model convergence, a Gaussian function is used to generate a joint prediction heatmap during model training, and mean squared error is used for coarse joint prediction. Specifically, the training process of the human pose estimation model further includes: determining a target loss function; and training the calibrated deep convolutional neural network based on the target loss function and the training set to obtain the human pose estimation model.

[0115] The target loss function is determined by a first loss function, a second loss function, a third loss function, and a fourth loss function. The first loss function represents the loss value between the predicted human joint point corresponding to the second feature map and the human joint point in the first image. The second loss function represents the loss value between the offset of the predicted human joint point corresponding to the second feature map and the average offset of the predicted human joint point. The third loss function represents the loss value between the distance between each predicted human joint point in the predicted human joint point corresponding to the fourth feature map and the distance between each predicted human joint point in the first image. The fourth loss function represents the loss value between the predicted human joint point corresponding to the fused feature map and the human joint point in the first image.

[0116] The first loss function is:

[0117] Among them, L h Let represent the first loss function value; i represents the human body number, j represents the human body joint number, and i and j are positive integers; P i (p j ) represents the predicted joint point of the i-th person in the second feature map; This represents a heatmap of the joints corresponding to the second feature map, with a scale of λ×96×72. This represents the coarse joint points in the joint point heatmap corresponding to the second feature map. Specifically, the position heatmap of the joint points corresponding to the second feature map is obtained, and the coarse position of the joint points is determined based on the position heatmap. λ represents the number of joint points; G i (p j ) represents the actual human joints of the i-th person in the first image; G = {g1,...,g λ} represents the heatmap of the joint points corresponding to the first image, g λ This represents the actual human joints in the joint point heatmap corresponding to the first image. Specifically, after the sample image is annotated, a ground truth heatmap corresponding to the sample image (first image) can be constructed using Gaussian activation.

[0118] The second loss function is:

[0119] Among them, L o Indicates the value of the second loss function; O i (p j ) represents the offset of the predicted human joint point of the i-th person in the second feature map. This represents the average offset of the predicted joint points of the i-th person in the second feature map. Specifically, the offset heatmap of the joint points corresponding to the second feature map is obtained, and the second loss function is determined based on the offset heatmap.

[0120] The third loss function is:

[0121] Among them, L b The value of the third loss function is represented; k and n represent the numbers of the human joints in the fourth feature map, and k,n∈j; This represents the k-th human joint predicted in the fourth feature map. This represents the nth human joint point predicted in the fourth feature map; This represents the actual k-th human body joint in the first image. This represents the actual nth human body joint in the first image; This represents the actual distance between adjacent human body joints. This represents the predicted distance between adjacent human joints. By calculating the third loss function based on the human skeleton, the network can be constrained to learn prior knowledge of human structure. Furthermore, this still refers to the prediction of joints for the i-th person, with subscripts in 2D. k and 2D n Let i represent the different joints of the i-th person.

[0122] The fourth loss function is:

[0123] Among them, L m M represents the value of the fourth loss function. i (m j ) represents the predicted human joint point of the i-th person in the fused feature map; This represents the joint heatmap corresponding to the fused feature map. G represents the coarse joints in the joint heatmap corresponding to the fused feature map, where λ represents the number of joints; i (p j ) represents the actual human joints of the i-th person in the first image; G = {g1,...,g λ} represents the heatmap of the joint points corresponding to the first image, g λ This represents the actual human joint in the joint point heatmap corresponding to the first image.

[0124] Furthermore, the process of determining the target loss function specifically includes: according to the formula L=αL h +βL o +γL b +δL m Determine the target loss function; where α represents the weight of the first loss function value, β represents the weight of the second loss function value, γ represents the weight of the third loss function value, and δ represents the weight of the fourth loss function value.

[0125] Preferably, in the joint position heatmap, the maximum pixel value position of each joint is found using a non-maximum suppression algorithm. The maximum pixel value position is used as the center point of the joint position, and the center point is expanded to generate a circle with a radius of R pixels. The values ​​within the radius are retained, and the remaining values ​​are set to 0 to determine the initial position of the center joint. In the joint offset heatmap, the maximum pixel value in the offset heatmap is found using a non-maximum suppression algorithm, i.e., the offset vector. The predicted joint position, i.e., the human joint prediction point, is obtained by adding the initial position of the joint to the offset vector of the joint. Finally, multiple human joint prediction points are connected according to the pre-calibrated skeleton order to form a complete human posture.

[0126] Example 2

[0127] like Figure 4 As shown, this embodiment provides a pose estimation system based on a human structure-guided learning network. The pose estimation system includes:

[0128] The image acquisition subsystem 101 is used to acquire a human posture image to be tested; the human posture image to be tested includes multiple human joint points.

[0129] The pose estimation subsystem 201 is used to input the human pose image to be measured into the human pose estimation model to obtain the human pose.

[0130] The human pose estimation model is trained based on a training set and a calibrated deep convolutional neural network. The training set includes multiple first images and corresponding label information for the first images. The label information consists of human joints in the first images. The human joints are used to construct the human pose. The calibrated deep convolutional neural network includes a stem network, a first backbone network, a second backbone network, a human structure-guided learning branch network, and a feature fusion and refinement network.

[0131] The stem network's input is used to input the training set; the stem network's output is connected to the input of the first backbone network; the stem network is used to extract features from the first image in the training set to determine a first feature map. The first output of the first backbone network is connected to the input of the second backbone network, and the second output of the first backbone network is connected to the input of the human structure guided learning branch network; the first backbone network is used to extract features from the first feature map to determine a second feature map; the output of the second backbone network is connected to the first input of the feature fusion and refinement network, and the second backbone network is used to extract features from the second feature map to determine a third feature map. The output of the human structure guided learning branch network is connected to the second input of the feature fusion and refinement network; the human structure guided learning branch network is used to perform human structure analysis on the second feature map to determine a fourth feature map. The feature fusion and refinement network is used to perform multi-scale feature fusion on the third and fourth feature maps, and to refine the pose of the fused feature map to determine the human pose.

[0132] The human structure-guided learning branch network includes an encoding subnetwork and a decoding subnetwork. The input of the encoding subnetwork is connected to the second output of the first backbone network, the output of the encoding subnetwork is connected to the input of the decoding subnetwork, and the output of the decoding subnetwork is connected to the second input of the feature fusion and refinement network. The encoding subnetwork includes multiple encoding modules connected in sequence, and the decoding subnetwork includes multiple decoding modules connected in sequence. Each encoding module corresponds one-to-one with a decoding module, and each encoding module is residually connected to a corresponding decoding module.

[0133] The encoding module includes a first Transformer module with memory and a downsampling module connected in sequence; the first Transformer module with memory includes a non-overlapping window self-attention mechanism network and a forward propagation network with memory units. The decoding module includes an upsampling module and a second Transformer module with memory connected in sequence; the second Transformer module with memory includes a non-overlapping window self-attention mechanism network and a forward propagation network with memory units.

[0134] Compared with the prior art, the present invention also has the following advantages:

[0135] (1) This invention constructs a human structure-guided learning branch network, whose predicted adaptive parameters can help the human pose estimation model learn more customized representations to estimate the pose of a specific input. The body cues contained therein can be effectively integrated to constrain joint positions and pose structure. Furthermore, the human structure-guided learner can also effectively learn adaptive parameters for different input images, thereby quickly adapting the pose model based on parsed information, making it a dynamic pose model.

[0136] (2) This application integrates the “concepts” in the low-resolution feature map with rich semantic information into the “pixels” of the high-resolution feature map with strong spatial information through a recursive feature fusion operation based on the self-attention mechanism. This makes the accuracy of each joint point of the human body more accurate and greatly improves the accuracy of pose estimation, providing a pose reference for behavior recognition, pedestrian re-identification, abnormal behavior detection, etc.

[0137] (3) By outputting the intermediate sample during the model training process, this application can reduce the variance of the data sample and accelerate the convergence speed of the model. The designed weighted bone loss function (third loss function) can well constrain the human skeleton, assist in the prediction of joint position and category, and further improve the accuracy of joint prediction.

[0138] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0139] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A pose estimation method based on a human structure-guided learning network, characterized in that, The attitude estimation method includes: Acquire an image of the human body posture to be tested; the image of the human body posture to be tested includes multiple human joint points; The human pose image to be tested is input into the human pose estimation model to obtain the human pose. The human pose estimation model is trained using a training set and a calibrated deep convolutional neural network. The training set includes multiple first images and corresponding label information for each first image. The label information consists of human joints in the first images. The human joints are used to construct the human pose. The calibrated deep convolutional neural network includes a stem network, a first backbone network, a second backbone network, a human structure-guided learning branch network, and a feature fusion and refinement network. The stem network's input is used to input the training set; the stem network's output is connected to the input of the first backbone network; the stem network is used to extract features from the first image in the training set to determine a first feature map. The first output of the first backbone network is connected to the input of the second backbone network, and the second output of the first backbone network is connected to the input of the human structure guided learning branch network; the first backbone network is used to extract features from the first feature map to determine the second feature map. The output of the second backbone network is connected to the first input of the feature fusion and refinement network. The second backbone network is used to extract features from the second feature map to determine the third feature map. The output of the human structure guided learning branch network is connected to the second input of the feature fusion and refinement network; the human structure guided learning branch network is used to perform human structure analysis on the second feature map to determine the fourth feature map. The feature fusion and refinement network is used to perform multi-scale feature fusion on the third feature map and the fourth feature map, and to refine the pose of the fused feature map to determine the human pose. The training process of the human pose estimation model also includes: Determine the target loss function; train the calibrated deep convolutional neural network according to the target loss function and the training set to obtain a human pose estimation model; The target loss function is determined by a first loss function, a second loss function, a third loss function, and a fourth loss function. The first loss function represents the loss value between the predicted human joint point corresponding to the second feature map and the human joint point in the first image. The second loss function represents the loss value between the offset of the predicted human joint point corresponding to the second feature map and the average offset of the predicted human joint point. The third loss function represents the loss value between the distance between each predicted human joint point in the predicted human joint point of the fourth feature map and the distance between each predicted human joint point in the first image. The fourth loss function represents the loss value between the predicted human joint point corresponding to the fused feature map and the human joint point in the first image. The first loss function is: Among them, L h Let represent the first loss function value; i represents the human body number, j represents the human body joint number, and i and j are positive integers; P i (p j ) represents the predicted joint point of the i-th person in the second feature map; This represents the heatmap of the joint points corresponding to the second feature map. G represents the coarse joints in the joint heatmap corresponding to the second feature map, where λ represents the number of joints; i (p j ) represents the actual human joints of the i-th person in the first image; G = {g1,...,g λ } represents the heatmap of the joint points corresponding to the first image, g λ This represents the actual human joint points in the joint point heatmap corresponding to the first image; The second loss function is: Among them, L o Indicates the value of the second loss function; O i (p j () represents the offset of the predicted human joint point of the i-th person in the second feature map. This represents the average offset of the predicted joint points of the i-th person in the second feature map. The third loss function is: Among them, L b The value of the third loss function is represented; k and n represent the numbers of the human joints in the fourth feature map, and k,n∈j; This represents the k-th human joint predicted in the fourth feature map. This represents the nth human joint point predicted in the fourth feature map; This represents the actual k-th human body joint in the first image. This represents the actual nth human body joint in the first image; This represents the actual distance between adjacent human body joints. This indicates the predicted distance between adjacent human body joints; The fourth loss function is: Among them, L m M represents the value of the fourth loss function. i (m j ) represents the predicted human joint point of the i-th person in the fused feature map; This represents the joint heatmap corresponding to the fused feature map. This refers to the coarse joint points in the joint heatmap corresponding to the fused feature map.

2. The pose estimation method based on a human structure-guided learning network according to claim 1, characterized in that, The human body structure-guided learning branch network includes an encoding subnetwork and a decoding subnetwork; The input of the encoding subnetwork is connected to the second output of the first backbone network, the output of the encoding subnetwork is connected to the input of the decoding subnetwork, and the output of the decoding subnetwork is connected to the second input of the feature fusion and refinement network. The encoding subnetwork includes multiple encoding modules connected in sequence, and the decoding subnetwork includes multiple decoding modules connected in sequence; the encoding module and the decoding module correspond one-to-one, and one encoding module is residually connected to one of the corresponding decoding modules; The encoding module includes a first memory-enabled Transformer module and a downsampling module connected in sequence; the first memory-enabled Transformer module includes a non-overlapping window self-attention mechanism network and a forward propagation network with memory units; The decoding module includes an upsampling module and a second memory-based Transformer module connected in sequence; the second memory-based Transformer module includes a non-overlapping window self-attention mechanism network and a forward propagation network with memory units.

3. The pose estimation method based on a human structure-guided learning network according to claim 2, characterized in that, The feature fusion and refinement network includes a multi-scale fusion module and a pose refinement module; The first input terminal of the multi-scale fusion module is connected to the output terminal of the second backbone network, the second input terminal of the multi-scale fusion module is connected to the output terminals of the multiple decoding modules respectively, and the output terminal of the multi-scale fusion module is connected to the input terminal of the pose refinement module. The output of the posture refinement module is used to output the human posture. The multi-scale fusion module includes a feature fusion submodule and multiple feature extraction submodules; The number of feature extraction sub-modules is the same as the number of decoding modules, and one feature extraction sub-module corresponds to one decoding module; The first input terminal of the feature extraction submodule is connected to the output terminal of the second backbone network; the second input terminal of the feature extraction submodule is connected to the output terminal of the corresponding decoding module; the output terminal of the feature extraction submodule is connected to the input terminal of the feature fusion submodule; and the output terminal of the feature fusion submodule is connected to the input terminal of the pose refinement module. The feature fusion submodule is used to fuse the sub-feature maps output by multiple feature extraction submodules to determine the final feature map; The posture refinement module is used to determine multiple human joints based on the final feature map, and to determine human posture based on the multiple human joints.

4. The pose estimation method based on a human structure-guided learning network according to claim 1, characterized in that, The process of determining the target loss function specifically includes: According to the formula L=αL h +βL o +γL b +δL m Determine the target loss function; Where α represents the weight of the first loss function value, β represents the weight of the second loss function value, γ represents the weight of the third loss function value, and δ represents the weight of the fourth loss function value.

5. The pose estimation method based on a human structure-guided learning network according to claim 1, characterized in that, The first backbone network includes a first subnet and a second subnet; The input of the first subnet is connected to the output of the stem network, and the output of the first subnet is connected to the input of the second subnet; the first output of the second subnet is connected to the input of the second backbone network, and the second output of the second subnet is connected to the input of the human structure guided learning branch network. The first subnet includes multiple residual modules connected in sequence; The second subnet includes a first branch and a second branch; both the first branch and the second branch include multiple residual modules, and the number of residual modules in the first branch is the same as the number of residual modules in the second branch. The input of the first branch is connected to the first output of the first subnet, the first output of the first branch is connected to the input of the second backbone network, and the second output of the first branch is connected to the input of the human structure guided learning branch network. The second branch is derived from the first subnet through downsampling; the input of the second branch is connected to the second output of the first subnet, the first output of the second branch is connected to the input of the second backbone network, and the second output of the second branch is connected to the input of the human structure guided learning branch network.

6. The pose estimation method based on a human structure-guided learning network according to claim 5, characterized in that, The second backbone network includes a third subnet and a fourth subnet; The first input terminal of the third subnet is connected to the first branch, and the second input terminal of the third subnet is connected to the second branch; the output terminal of the third subnet is connected to the input terminal of the fourth subnet, and the output terminal of the fourth subnet is connected to the first input terminal of the feature fusion and refinement network. The third subnet includes parallel third, fourth, and fifth branches; each of the third, fourth, and fifth branches includes multiple residual modules, and the number of residual modules in the third branch, the fourth branch, and the fifth branch are all the same. The input terminal of the third branch is connected to the first output terminal of the first branch and the first output terminal of the second branch, respectively; the input terminal of the fourth branch is connected to the first output terminal of the first branch and the first output terminal of the second branch, respectively. The fifth branch is derived from the second branch through downsampling; the input of the fifth branch is connected to the first output of the first branch and the first output of the second branch, respectively. The fourth subnet includes parallel sixth, seventh, eighth and ninth branches; each of the sixth, seventh, eighth and ninth branches includes multiple residual modules, and the number of residual modules in the sixth branch, the seventh branch, the eighth branch and the ninth branch are all the same. The input terminal of the sixth branch is connected to the output terminals of the third branch, the fourth branch, and the fifth branch, respectively; the input terminal of the seventh branch is connected to the output terminals of the third branch, the fourth branch, and the fifth branch, respectively; the input terminal of the eighth branch is connected to the output terminals of the third branch, the fourth branch, and the fifth branch, respectively. The ninth branch is derived from the fifth branch through downsampling; the input of the ninth branch is connected to the output of the third branch, the output of the fourth branch, and the output of the fifth branch, respectively.

7. A pose estimation system based on a human structure-guided learning network, employing the pose estimation method based on a human structure-guided learning network as described in any one of claims 1-6, characterized in that, The attitude estimation system includes: An image acquisition subsystem is used to acquire an image of the human body posture to be tested; the image of the human body posture to be tested includes multiple human joint points. The pose estimation subsystem is used to input the human pose image to be measured into the human pose estimation model in order to obtain the human pose. The human pose estimation model is trained using a training set and a calibrated deep convolutional neural network. The training set includes multiple first images and corresponding label information for each first image. The label information consists of human joints in the first images. The human joints are used to construct the human pose. The calibrated deep convolutional neural network includes a stem network, a first backbone network, a second backbone network, a human structure-guided learning branch network, and a feature fusion and refinement network. The stem network's input is used to input the training set; the stem network's output is connected to the input of the first backbone network; the stem network is used to extract features from the first image in the training set to determine a first feature map. The first output of the first backbone network is connected to the input of the second backbone network, and the second output of the first backbone network is connected to the input of the human structure guided learning branch network; the first backbone network is used to extract features from the first feature map to determine the second feature map. The output of the second backbone network is connected to the first input of the feature fusion and refinement network. The second backbone network is used to extract features from the second feature map to determine the third feature map. The output of the human structure guided learning branch network is connected to the second input of the feature fusion and refinement network; the human structure guided learning branch network is used to perform human structure analysis on the second feature map to determine the fourth feature map. The feature fusion and refinement network is used to perform multi-scale feature fusion on the third feature map and the fourth feature map, and to refine the pose of the fused feature map to determine the human pose.

8. The pose estimation system based on a human structure-guided learning network according to claim 7, characterized in that, The human body structure-guided learning branch network includes an encoding subnetwork and a decoding subnetwork; The input of the encoding subnetwork is connected to the second output of the first backbone network, the output of the encoding subnetwork is connected to the input of the decoding subnetwork, and the output of the decoding subnetwork is connected to the second input of the feature fusion and refinement network. The encoding subnetwork includes multiple encoding modules connected in sequence, and the decoding subnetwork includes multiple decoding modules connected in sequence; the encoding module and the decoding module correspond one-to-one, and one encoding module is residually connected to one of the corresponding decoding modules; The encoding module includes a first memory-enabled Transformer module and a downsampling module connected in sequence; the first memory-enabled Transformer module includes a non-overlapping window self-attention mechanism network and a forward propagation network with memory units; The decoding module includes an upsampling module and a second memory-based Transformer module connected in sequence; the second memory-based Transformer module includes a non-overlapping window self-attention mechanism network and a forward propagation network with memory units.

Images