Human pose estimation method, device, equipment and storage medium

Abstract

A human posture estimation method, device, equipment and storage medium are disclosed, and belong to the technical field of image processing. The method comprises the following steps: acquiring a first data set; training a three-dimensional human modeling model by using the first data set; and performing human posture estimation based on a three-dimensional human model output by the trained three-dimensional human modeling model. The three-dimensional human modeling model comprises a ResNet feature extractor, an HMR regressor and an SMPL model connected in sequence. The input of the HMR regressor comprises image features of a first image and boundary box geometric features of a second human body. The output of the HMR regressor comprises joint point features. The boundary box geometric features are used to convert the joint point features to a first coordinate system, so as to calculate a projection loss of the joint point features in the first coordinate system. The first coordinate system is a coordinate system of an original camera. The method can accurately realize human posture estimation on an image with slight posture changes.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a method, apparatus, device and storage medium for human pose estimation. Background Technology

[0002] Human pose estimation refers to the process of estimating the pose of the human body in an input image to obtain the pose information of the human body in the image.

[0003] In related technologies, human pose estimation methods include: inputting an image containing a human body into a classic HMR framework to obtain a 3D human body model, and then estimating the human pose based on the 3D model. The classic HMR framework consists of a convolutional neural network, an HMR regressor, and a standard SMPL (Skinned Multi-Person Linear Model) model connected in sequence.

[0004] However, the 3D human body model output by the classic HMR framework is not sensitive to pose changes, which leads to errors when using the above method to estimate human pose in images with slight pose changes. Summary of the Invention

[0005] This disclosure provides a method, apparatus, device, and storage medium for human pose estimation, which can accurately estimate human pose from images with slight pose changes. The technical solution includes at least the following:

[0006] A first aspect provides a method for estimating human pose, comprising: acquiring a first dataset, the first dataset including multiple images of a first human body; training a 3D human body modeling model using the first dataset, the trained 3D human body modeling model being used to generate a 3D human body model based on a single image input to the 3D human body modeling model; and performing human pose estimation based on the 3D human body model output by the trained 3D human body modeling model; wherein the 3D human body modeling model includes a ResNet feature extractor, a Hybrid Multimodal Representation (HMR) regressor, and a skinned multi-person linear SMPL model connected in sequence, the input of the HMR regressor including image features of a first image and bounding box geometric features of a second human body, the output of the HMR regressor including joint features, the first image being an image input to the 3D human body modeling model, the second human body being the human body in the first image, the joint features including multiple joint coordinates, the bounding box geometric features being used to transform the joint features to a first coordinate system to calculate the projection loss of the joint features in the first coordinate system, the first coordinate system being the coordinate system of the original camera, and the original camera being the camera corresponding to the first image.

[0007] Optionally, the images of the first human body in the first dataset include depth RGB images of the first human body at azimuth angles of 0°, 60°, 120°, 180°, 240°, 300°, and 360°, respectively. The ResNet feature extractor includes six branches, each branch being used to extract feature vectors from the depth RGB image at one of the azimuth angles. Training the 3D human body modeling model using the first dataset includes: pre-training the six branches of the ResNet feature extractor based on the depth RGB images at different azimuth angles in the first dataset; freezing the parameters of the six branches of the ResNet feature extractor after pre-training, while fine-tuning the parameters of the HMR regressor; after fine-tuning the parameters of the HMR regressor, optimizing the parameters of the HMR regressor using total error loss to train the 3D human body modeling model, wherein the total error loss includes the projection loss of the joint features calculated on the first coordinate system.

[0008] Optionally, optimizing the parameters of the HMR regressor using the total error loss includes: establishing a projection chain relationship based on a first root displacement, wherein the projection chain relationship indicates the process of transforming the 3D coordinates in the HMR coordinate system to the 3D coordinates in the first coordinate system, and projecting the 3D coordinates in the first coordinate system onto a 2D panoramic image, wherein the HMR coordinate system is the coordinate system of the HMR virtual camera, the HMR virtual camera is the camera used to annotate the bounding box corresponding to the second human body, the first root displacement is the root displacement between the HMR coordinate system and the first coordinate system, and the first root displacement is determined based on the geometric features of the bounding box of the second human body; obtaining the first joint feature of the second human body, wherein the first joint feature includes multiple joint coordinates of the second human body predicted by the HMR regressor, and the first joint feature is a feature in the HMR coordinate system; projecting the first joint feature onto the 2D panoramic image using the projection chain relationship; calculating the 2D reprojection loss of the first joint feature; determining the total error loss based on the 2D reprojection loss; and optimizing the parameters of the HMR regressor based on the total error loss to train the 3D human body modeling model.

[0009] Optionally, the bounding box geometry of the second human body is represented by the following formula:

[0010]

[0011] in, The bounding box geometry features, The focal length of the original camera; This indicates the coordinates of the center of the bounding box used to annotate the second human figure in the image coordinate system, which is a two-dimensional coordinate system with the center of the first image as its origin. This indicates the side length of the bounding box used to annotate the second human body. The width of the first image. The height is the height of the first image.

[0012] Optionally, the first displacement can be obtained using the following formula:

[0013]

[0014] in, This represents the root displacement of the HMR coordinate system relative to the first coordinate system. The translation parameters for weak perspective projection in the HMR coordinate system are: , The focal length of the HMR virtual camera. The resolution used to annotate the bounding box of the second human body. This is a proportional parameter.

[0015] Optionally, the ResNet feature extractor and the HMR regressor are connected via a dual-pooling channel attention mechanism, which includes a global average pooling channel and a global max pooling channel. The global average pooling channel and the global max pooling channel are concatenated through the attention mechanism. The method further includes: inputting the feature vectors output from the six branches of the ResNet feature extractor into the dual-pooling channel attention mechanism to obtain the image features of the first image; concatenating the image features of the first image with the bounding box geometric features to obtain a first fused feature, which is used as input to the HMR regressor.

[0016] Secondly, a human pose estimation device is also provided, comprising: an acquisition module for acquiring a first dataset, the first dataset including multiple images of a first human body; a training module for training a 3D human body modeling model using the first dataset, the trained 3D human body modeling model being used to generate a 3D human body model based on a single image input to the 3D human body modeling model; and a human pose estimation module for performing human pose estimation based on the 3D human body model output by the trained 3D human body modeling model; wherein the 3D human body modeling model includes a ResNet feature extractor, a Hybrid Multimodal Representation (HMR) regressor, and a skinned multi-person linear SMPL model connected in sequence, the input of the HMR regressor including image features of a first image and bounding box geometric features of a second human body, the output of the HMR regressor including joint features, the first image being the image input to the 3D human body modeling model, the second human body being the human body in the first image, the joint features including multiple joint coordinates, the bounding box geometric features being used to transform the joint features to a first coordinate system to calculate the projection loss of the joint features in the first coordinate system, the first coordinate system being the coordinate system of the original camera, and the original camera being the camera corresponding to the first image.

[0017] Optionally, the images of the first human body in the first dataset include depth RGB images of the first human body at azimuth angles of 0°, 60°, 120°, 180°, 240°, 300°, and 360°, respectively. The ResNet feature extractor includes six branches, each branch being used to extract a feature vector from the depth RGB image at one of the azimuth angles. The training module is further used to pre-train the six branches of the ResNet feature extractor based on the depth RGB images at different azimuth angles in the first dataset. After the pre-training of the six branches of the ResNet feature extractor is completed, the parameters of the six branches of the ResNet feature extractor are frozen, while the parameters of the HMR regressor are fine-tuned. After fine-tuning the parameters of the HMR regressor, the parameters of the HMR regressor are optimized using total error loss to train the three-dimensional human body modeling model. The total error loss includes the projection loss of the joint features calculated on the first coordinate system.

[0018] Optionally, the training module is further configured to establish a projection chain relationship based on a first root displacement, wherein the projection chain relationship indicates the process of transforming the 3D coordinates in the HMR coordinate system to the 3D coordinates in the first coordinate system, and projecting the 3D coordinates in the first coordinate system onto a 2D panoramic image. The HMR coordinate system is the coordinate system of the HMR virtual camera, and the HMR virtual camera is the camera used to annotate the bounding box corresponding to the second human body. The first root displacement is the root displacement between the HMR coordinate system and the first coordinate system, and the first root displacement is determined based on the geometric features of the bounding box of the second human body. The module also acquires the first joint feature of the second human body, which includes multiple joint coordinates of the second human body predicted by the HMR regressor. The first joint feature is a feature in the HMR coordinate system. The module uses the projection chain relationship to project the first joint feature onto the 2D panoramic image. The module calculates the 2D reprojection loss of the first joint feature. Based on the 2D reprojection loss, the module determines the total error loss. Based on the total error loss, the module optimizes the parameters of the HMR regressor and trains the 3D human body modeling model.

[0019] Optionally, the ResNet feature extractor and the HMR regressor are connected via a dual-pooling channel attention mechanism, which includes a global average pooling channel and a global max pooling channel. The global average pooling channel and the global max pooling channel are concatenated through the attention mechanism. The training module is further configured to input the feature vectors output from the six branches of the ResNet feature extractor into the dual-pooling channel attention mechanism to obtain the image features of the first image; and to concatenate the image features of the first image with the bounding box geometric features to obtain a first fused feature, which is used as input to the HMR regressor.

[0020] Thirdly, a computer device is also provided, comprising: a memory and a processor, wherein the memory stores at least one computer program, the at least one computer program being loaded and executed by the processor to perform the human pose estimation method described in the above embodiments.

[0021] Fourthly, a computer-readable storage medium is also provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to perform the human pose estimation method described in the above embodiments.

[0022] Fifthly, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the method described in the first aspect.

[0023] The beneficial effects of the technical solutions provided in this disclosure include at least the following:

[0024] When projecting coordinates in three-dimensional space onto a two-dimensional plane, the original camera is more sensitive to slight changes in human posture compared to an HMR virtual camera. Therefore, in this embodiment, the three-dimensional human body modeling model includes bounding box geometric features. These features are used to map the joint coordinates predicted by the HMR regressor onto the first coordinate system of the original camera, so as to calculate the projection loss of the joint features in the first coordinate system. That is, when calculating the projection loss, the model projects the data into the first coordinate system before calculating the projection loss. In this way, the three-dimensional human body modeling model is more sensitive to slight changes in human posture, thereby improving the accuracy of the three-dimensional human body modeling model in recognizing slight posture changes. Based on this three-dimensional human body modeling model, human posture estimation can be performed more accurately on images with slight posture changes. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating a human pose estimation method provided in an exemplary embodiment of this disclosure is shown;

[0027] Figure 2 This is a schematic diagram of the projection of human joints in different poses onto a two-dimensional plane by the HMR virtual camera and the original camera.

[0028] Figure 3 A flowchart of a human pose estimation method provided by another exemplary embodiment of this disclosure is shown;

[0029] Figure 4 A schematic diagram illustrating the geometric meaning of the bounding box geometric features;

[0030] Figure 5 A schematic diagram of the structure of the trained 3D human body model;

[0031] Figure 6 This is a schematic diagram of a single image input to a 3D human body model and the 3D human body model output from the 3D human body model.

[0032] Figure 7 A schematic diagram of the structure of a human pose estimation device provided in an exemplary embodiment of this disclosure is shown;

[0033] Figure 8This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0034] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” “third,” and similar terms used in this patent application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising” or “including” and similar terms mean that the elements or objects preceding “comprising” or “including” encompass the elements or objects listed following “comprising” or “including” and their equivalents, and do not exclude other elements or objects. The terms “connected” or “linked” and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0035] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0036] Figure 1 A flowchart illustrating a human pose estimation method provided in an exemplary embodiment of this disclosure is shown, which can be executed by a computer device. See also Figure 1 The method includes:

[0037] In step 101, the first dataset is obtained.

[0038] The first dataset includes multiple images of the first human body.

[0039] Here, the first dataset is the dataset used to train the 3D human body modeling model, and the first human body can be any human body.

[0040] Optionally, the image of the first human body in the first dataset is a depth RGB image. In this case, the first dataset is obtained by following three steps.

[0041] The first step is to acquire multiple RGB images of the first human body.

[0042] These multiple RGB images were taken using an RGB camera. The RGB color mode is a color standard that produces a wide variety of colors by varying the red, green, and blue color channels and superimposing these channels.

[0043] The second step involves using an RGBD sensor to capture images of the first human body, resulting in multiple RGBD images.

[0044] An RGBD (RGB Depth) sensor is a sensor that contains depth information, such as distance. RGBD images captured by an RGBD sensor contain depth information.

[0045] The third step is to align the depth information in the RBGD image to the RGB image to obtain the enhanced RGB image.

[0046] Optionally, the depth information in the RBGD image is aligned to the RGB image, including: using the ICP (Iterative Closest Point) algorithm to achieve spatial registration between the RGB image and the RBGD image; applying the same geometric transformation matrix to the RGB image and the RBGD image to maintain the projection relationship; and finally using a hole-filling algorithm based on Conditional Random Field (CRF) to repair invalid depth regions.

[0047] The process of aligning the depth information in an RGB image to an RGB image is essentially the process of enhancing the RGB image to give it depth information.

[0048] One related technology uses RGBD cameras to acquire depth RGB images. However, existing RGBD cameras (such as Kinect) have a fixed depth resolution within a certain range, and the accuracy of the depth information may not be ideal at long distances or in complex environments.

[0049] By employing steps one through three above to acquire depth RGB images, the precision of the RGB camera and RGBD sensor can be flexibly adjusted. For example, if the RGB camera's precision is insufficient, a higher-precision RGB camera can be replaced; or if the RGBD sensor's precision is insufficient, a higher-precision RGBD sensor can be selected. This allows for the satisfaction of different precision requirements for depth RGB images.

[0050] Alternatively, the images in the first dataset can be enhanced by: horizontally flipping the images, vertically flipping the images, randomly scaling the images, where the scaling ratio ranges from 0.85 to 1.25, and randomly rotating the images, where the rotation ranges from -20° to 20°.

[0051] In step 102, a 3D human body modeling model is trained using the first dataset.

[0052] The trained 3D human body modeling model is used to generate a 3D human body model from a single image input to it. A conventional HMR framework can output a 3D human body model from a single image. This embodiment of the disclosure improves the HMR framework, which is the 3D human body modeling model itself; therefore, the 3D human body modeling model can generate a 3D human body model from a single image input to it.

[0053] The 3D human modeling model comprises a ResNet (Residual Network) feature extractor, a Hybrid Multimodal Representation (HMR) regressor, and a skinned multi-person linear SMPL model, connected sequentially. The input to the HMR regressor includes image features from a first image and bounding box geometric features of the second human body. The output of the HMR regressor includes joint features. The first image is the image input to the 3D human modeling model, and the second human body is the human body in the first image. The joint features include multiple joint coordinates. The bounding box geometric features are used to transform the joint features to a first coordinate system to calculate the projection loss of the joint features in the first coordinate system. The first coordinate system is the coordinate system of the original camera, and the original camera is the camera corresponding to the first image.

[0054] Here, the first image is the image input into the 3D human body modeling model, and the first image includes at least one human body. That is, during the training and testing of the 3D human body modeling model, the first image is the image in the first dataset. When performing human pose estimation based on the 3D human body model output from the trained 3D human body modeling model, the first image is an image outside the first dataset (e.g., the image for which human pose estimation is required). Similarly, the second human body is the human body in the first image that needs to be modeled in 3D. During the training and testing of the 3D human body modeling model, the second human body is the first human body; when performing human pose estimation based on the 3D human body model output from the trained 3D human body modeling model, the second human body is the human body other than the first human body.

[0055] Here, the original camera refers to the camera used to capture the first image. This original camera is located on a straight line that passes through the center of the first image and is perpendicular to the first image. The distance between the original camera and the center of the first image is the focal length of the original camera.

[0056] In normal circumstances, HMR regressors do not involve the original camera's coordinate system when calculating projection loss. Instead, they generate an HMR virtual camera for the cropped image, perform predictions based on the HMR coordinate system of this virtual camera, and directly calculate the projection loss based on the prediction results. This HMR virtual camera lies on a straight line passing through the center of the cropped image (i.e., the bounding box) and perpendicular to the cropped image. The distance between the HMR virtual camera and the center of the cropped image is the focal length of the HMR virtual camera.

[0057] When projecting coordinates from three-dimensional space onto a two-dimensional plane, the original camera is more sensitive to subtle changes in human posture compared to an HMR virtual camera. The following section combines... Figure 2 This process will be explained.

[0058] Figure 2 This is a schematic diagram of the projection of human joints in different poses onto a two-dimensional plane by the HMR virtual camera and the original camera. Figure 2 Part (a) is a schematic diagram of the first and second postures. For example... Figure 2 As shown in part (a), the first pose 201 is the pose of the human body facing the HMR virtual camera, and the second pose 202 is the pose of the human body after rotating 30 degrees clockwise from the first pose 201. In the front view, the difference between the first pose 201 and the second pose 202 is basically only the width of the line connecting the two joints at the shoulders. In the top view, it can be seen that there is an angular difference between the first pose 201 and the second pose 202. This difference is quite subtle (only a 30-degree rotation).

[0059] When an HMR regressor performs regression, there is inevitably a process of projecting the predicted three-dimensional coordinates of human joints onto a two-dimensional plane. Figure 2 Part (b) is a schematic diagram showing the projection of the human joints of the first pose 201 and the second pose 202 onto a two-dimensional plane by the HMR virtual camera and the original camera, respectively. Figure 2 As shown in part (b), the red dashed line represents the projection process of the first posture 201, and the red solid line represents the projection of the first posture 201 onto the two-dimensional plane 205; the blue dashed line represents the projection process of the second posture 202, and the blue solid line represents the projection of the second posture 202 onto the two-dimensional plane 205.

[0060] here, Figure 2 In part (b), it is assumed that the second human body is located in the left half of the first image, so the HMR virtual camera 203 is to the left of the original camera 204.

[0061] It can be seen that when projecting in the HMR coordinate system of the HMR virtual camera 203, the projected line segment 'a' corresponding to the shoulder joints of the first pose 201 and the projected line segment 'b' corresponding to the shoulder joints of the second pose 202 are quite close. This makes the HMR virtual camera insensitive to slight pose changes, easily identifying the first pose 201 and the second pose 202 as the same pose, resulting in the generation of two identical 3D human models. However, when projecting in the first coordinate system of the original camera 204, there is a significant difference between the projected line segment 'c' corresponding to the shoulder joints of the first pose 201 and the projected line segment 'd' corresponding to the shoulder joints of the second pose 202; that is, the difference between 'c' and 'd' is greater than the difference between 'a' and 'b'. In other words, in the first coordinate system of the original camera 204, the differences caused by slight pose changes are more pronounced, or rather, the first coordinate system of the original camera 204 amplifies the differences caused by slight pose changes.

[0062] Therefore, compared to the HMR coordinate system of the HMR virtual camera, the original camera's first coordinate system is more sensitive to slight changes in human posture, thus enabling more accurate identification of different postures with only minor variations. In other words, when projecting the three-dimensional coordinates in the first coordinate system onto a two-dimensional plane, the spatial offset of the shoulder bone points can be represented more precisely, significantly improving viewpoint sensitivity and compensating for deficiencies in geometric priors.

[0063] In this way, by inputting the bounding box geometric features into the HMR regressor, the HMR regressor can understand the difference between the HMR coordinate system and the first coordinate system when predicting joint features. The bounding box geometric features are also used to map the joint coordinates predicted by the HMR regressor onto the original camera's first coordinate system, so as to calculate the projection loss of the joint features in the first coordinate system. That is, when calculating the projection loss, projection is performed in the first coordinate system. Thus, the 3D human modeling model is more sensitive to slight changes in human posture, thereby improving the accuracy of the 3D human modeling model in recognizing slight posture changes. In other words, this 3D human modeling model can accurately construct a 3D human body model.

[0064] In step 103, human pose estimation is performed based on the 3D human model output from the trained 3D human modeling model.

[0065] After the 3D human body modeling model is trained, it can reconstruct a relatively accurate 3D human body model based on a single image. Based on this accurate 3D human body model, human pose estimation can be performed accurately.

[0066] When projecting coordinates in three-dimensional space onto a two-dimensional plane, the original camera is more sensitive to slight changes in human posture compared to an HMR virtual camera. Therefore, in this embodiment, the three-dimensional human body modeling model includes bounding box geometric features. These features are used to map the joint coordinates predicted by the HMR regressor onto the first coordinate system of the original camera, so as to calculate the projection loss of the joint features in the first coordinate system. That is, when calculating the projection loss, the model projects the data into the first coordinate system before calculating the projection loss. In this way, the three-dimensional human body modeling model is more sensitive to slight changes in human posture, thereby improving the accuracy of the three-dimensional human body modeling model in recognizing slight posture changes. Based on this three-dimensional human body modeling model, human posture estimation can be performed more accurately on images with slight posture changes.

[0067] Figure 3 A flowchart illustrating a human pose estimation method provided in another exemplary embodiment of this disclosure is shown, which can be performed by a computer device. See also Figure 3 The method includes:

[0068] In step 301, the first dataset is obtained.

[0069] For details on obtaining the first dataset, please refer to step 101 above.

[0070] Based on the multiple enhanced RGB images from step 101 above, a model of the first human body in the RGB images can be created.

[0071] While RGB images captured by an RGB camera have a limited field of view, they can be used to accurately model a first human body, resulting in a 3D human body model. Then, a virtual camera can be used to photograph this 3D model, and the resulting RGB image can be stored in the first dataset (image enhancement processing, such as integrating depth information into the RGB image, is also required). Since the virtual camera can be set to any angle and focal length, theoretically, RGB images of the first human body can be obtained from any angle.

[0072] In this case, step 301 further includes: constructing a virtual camera array; using virtual cameras in the virtual camera array to perform multi-view rendering and photography of the three-dimensional human model of the first human body, generating a synthesized RGB image and corresponding pose information.

[0073] The virtual camera array's parameter configuration includes: an azimuth angle ranging from 0° to 360°, with one virtual camera positioned every 60° within this range; within any given row of virtual cameras, the pitch angle varies from -30° to 60°, with one virtual camera positioned every 10° of pitch angle. Furthermore, the focal length of any virtual camera in the array can randomly vary within a range of 35mm ± 20%.

[0074] In this way, a first dataset can be constructed, which includes multiple images of the first human body, each of which is a depth RGB image. When divided according to azimuth angle, the images of the first human body in the first dataset include depth RGB images of the first human body at azimuth angles of 0°, 60°, 120°, 180°, 240°, 300°, and 360° respectively.

[0075] Optionally, corresponding to the six azimuth angles, the ResNet feature extractor has six branches, each of which is a ResNet-50 network. Each branch is used to extract the feature vector of the depth RGB image for a given azimuth angle. During the training of the 3D human modeling model using the first dataset, these six branches are used to extract features from the first human image at azimuth angles of 0°, 60°, 120°, 180°, 240°, 300°, and 360°, respectively. After the ResNet feature extractor is trained, it demonstrates good feature extraction capabilities for images at different angles.

[0076] In step 302, based on the depth RGB images at different azimuth angles in the first dataset, the six branches of the ResNet feature extractor are pre-trained.

[0077] Before proceeding to step 302, each part of the 3D human body modeling model will be described first. The 3D human body modeling model consists of a ResNet feature extractor, an HMR regressor, and an SMPL model connected in sequence.

[0078] The ResNet feature extractor consists of 6 branches, and the input to the ResNet feature extractor is the first image. With the ResNet feature extractor and the HMR regressor connected via a dual-channel pooling attention mechanism, the output of the ResNet feature extractor is 6 feature vectors extracted from the first image by the 6 branches. These 6 feature vectors are then processed by the dual-channel pooling attention mechanism to obtain the image features of the first image.

[0079] The input to the HMR regressor includes image features of the first image and geometric features of the bounding box of the second human body. The output of the HMR regressor is the joint features.

[0080] The input to the SMPL model is joint features, and the output of the SMPL model is a 3D human body model of the second human body.

[0081] In this embodiment, the ResNet feature extractor is first pre-trained. After the ResNet feature extractor is pre-trained, the parameters of the ResNet feature extractor are frozen and the parameters of the HMR regressor are fine-tuned. After the parameters of the HMR regressor are fine-tuned, the parameters of the HMR regressor are finally optimized using the total error loss, thereby realizing the training of the 3D human body modeling model.

[0082] In this case, optionally, the loss function of the six branches of the pre-trained ResNet feature extractor is expressed by Equation (1).

[0083] (1)

[0084] In formula (1), This refers to the loss function used for each branch when pre-training the ResNet feature extractor. For the first The feature vectors extracted from the branches corresponding to each azimuth angle. ,in The height of the feature vector. The width of the feature vector. Number of channels For the first The feature vectors extracted from the branches corresponding to each azimuth angle. It is an integer. The value range is from 1 to 6, where the first azimuth, the second azimuth, ..., the six azimuths represent 0°, 60°, ..., 360° respectively. Indicates except the first Any one of the other five azimuth angles besides the first one. Indicates the first The feature vectors extracted from the branches corresponding to each azimuth angle. Indicates calculation and Temperature-scaled cosine similarity between them Indicates calculation and Temperature-scaled cosine similarity between them.

[0085] For example, and The temperature scaling cosine similarity between the two is calculated using formula (2).

[0086] (2)

[0087] In formula (2), The temperature scaling parameter is a learnable parameter, initially... It can be set to 0.07. The meanings of the other parameters in formula (2) are the same as those in formula (1), and will not be detailed here.

[0088] Through formula (1), the features of adjacent viewpoints ( and It maintains maximum mutual information in the cosine similarity space while pushing away the correlation between features from non-adjacent viewpoints. Therefore, a pre-trained ResNet feature extractor can effectively solve the viewpoint sensitivity problem in single-viewpoint feature extraction, ensuring that the extracted features have geometric consistency.

[0089] When using formula (1) as the loss function to pre-train the ResNet feature extractor, the parameters of the ResNet feature extractor can be optimized by using gradient descent algorithm and Adam optimizer.

[0090] After the six branches of the ResNet feature extractor have been pre-trained, step 303 can be executed.

[0091] In step 303, the parameters of the six branches of the ResNet feature extractor are frozen, while the parameters of the HMR regressor are fine-tuned.

[0092] After pre-training the multi-view feature extraction network, the convolutional layer parameters of the six branches of the ResNet feature extractor can be frozen (except for the batch normalization layer, which can be updated). Then, the initial learning rate can be used. The Adam optimizer fine-tunes the parameters of the HMR regressor and implements an exponential decay strategy of 2% per training cycle, i.e., the decay formula is: .

[0093] Optionally, in the 3D human body modeling model, the ResNet feature extractor and the HMR regressor are connected through a dual-pooling channel attention mechanism, which includes a global average pooling channel and a global max pooling channel. The global average pooling channel and the global max pooling channel are concatenated through the attention mechanism.

[0094] In this case, step 303 may optionally include the following two steps.

[0095] The first step is to input the feature vectors output from the six branches of the ResNet feature extractor into a dual-channel pooling attention mechanism to obtain the image features of the first image;

[0096] The first step can be represented by formulas (3) to (4).

[0097] (3)

[0098] (4)

[0099] In formula (3), for The weight, Indicates to Perform global average pooling. , Indicates to Perform global max pooling. , The weight matrix for the dual-channel pooling attention mechanism is a learnable parameter. . This indicates a concatenation operation along the channel dimension. It is the sigmoid activation function.

[0100] In formula (4), The image features of the first image are defined as follows. The meanings of the other parameters in formulas (3) and (4) are the same as those in formula (1) above, and are omitted here for detailed explanation.

[0101] The second step is to concatenate the image features of the first image with the geometric features of the bounding box to obtain the first fused feature.

[0102] The first fusion feature is used as input to the HMR regressor. Based on this first fusion feature, the HMR regressor can be fine-tuned.

[0103] In this case, the input of this HMR regressor can be represented by formula (5).

[0104] (5)

[0105] In formula (5), These are the parameters input into the HMR regressor. This is the weight matrix of the HMR regressor. For bounding box geometric features, The first fusion feature, This is the bias of the HMR regressor. , All of these are learnable parameters. The meanings of the other parameters in formula (5) are the same as those in formula (4), and will not be elaborated here.

[0106] Bounding box geometry features It can be expressed using formula (6).

[0107] (6)

[0108] In formula (6), For bounding box geometric features, This refers to the original camera's focal length. This represents the coordinates of the center of the bounding box used to annotate the second human figure in the image coordinate system, which is a two-dimensional coordinate system with the center of the first image as its origin. This indicates the side length of the bounding box used to annotate the second human figure. The width of the first image. The height of the first image.

[0109] Figure 4 This is a schematic diagram illustrating the geometric meaning of the bounding box's geometric features. The following section combines... Figure 4 The parameters in the bounding box geometry feature are explained.

[0110] like Figure 4 As shown, the width of the first image 401 is , length is The image coordinate system xO1y takes the center O1 of the first image 401 as its origin.

[0111] The bounding box 402 used to annotate the second human body has its center at O2. Bounding box 402 is a square bounding box with a side length of b. In the image coordinate system xO1y, the coordinates of the center O2 of bounding box 402 can be expressed as: .

[0112] In the HMR coordinate system 403, O3 is the origin of the HMR coordinate system 403, and the position of O3 is the position of the HMR virtual camera. O3O2 is a straight line perpendicular to the bounding box 402, and this straight line passes through the center O2 of the bounding box 402. The length of O3O2 is the focal length of the HMR virtual camera.

[0113] In the first coordinate system 404, O4 is the origin of the first coordinate system 404, and the position of O4 is the position of the original camera. O4O1 is a straight line perpendicular to the first image 401, and this line passes through the center O1 of the first image 401. The length of O4O1 is the focal length of the original camera. The value of .

[0114] In bounding box geometry features middle, The first two items and It has geometric significance in terms of angle. As shown in the figure. Represents angle The tangent of O4O1 Represents angle The tangent value of O4O1. These two values ​​help the HMR regressor understand the angular relationship between the coordinates in bounding box 402 and the original camera coordinates. The third item and the fourth item It can reflect the proportion of bounding box 402 in the first image (in the third and fourth items) (It plays a normalization role), so that in multi-resolution scenarios (such as images from different cameras or different video streams), the HMR regressor can better understand the proportion of people in the image and learn additional global scale information.

[0115] In some cases, the focal length of the original camera If the truth value is known, then we can... China directly adopts The truth value. And the focal length of the original camera. When the truth value is unknown, then The original camera's focal length can be used. The approximate estimate is expressed by formula (7).

[0116] (7)

[0117] The meaning of the parameters in formula (7) is the same as that of the parameters in formula (6), and will not be elaborated here.

[0118] By fine-tuning the HMR regressor in step 303, a preliminary set of HMR regressor parameters can be obtained. At this point, the HMR regressor can initially understand the relationship between the original camera and the HMR virtual camera. However, the key points predicted by the HMR regressor are still coordinates in the HMR coordinate system. It is necessary to transform the coordinates predicted by the HMR regressor in the HMR coordinate system to the first coordinate system, and then further optimize the parameters of the HMR regressor based on step 304, that is, based on the coordinates in the first coordinate system.

[0119] In step 304, the parameters of the HMR regressor are optimized using the total error loss.

[0120] The total error loss includes the projection loss of the joint features calculated on the first coordinate system.

[0121] Optionally, step 304 includes the following step af.

[0122] Step a: Based on the first displacement, establish a projection chain relationship.

[0123] The projection chain relationship is used to indicate the process of transforming the three-dimensional coordinates in the HMR coordinate system to the three-dimensional coordinates in the first coordinate system, and projecting the three-dimensional coordinates in the first coordinate system onto the two-dimensional panoramic image. The HMR coordinate system is the coordinate system of the HMR virtual camera, and the HMR virtual camera is the camera corresponding to the bounding box used to annotate the second human body. The first root displacement is the root displacement between the HMR coordinate system and the first coordinate system, and the first displacement is determined based on the geometric features of the bounding box of the second human body.

[0124] Optionally, the first displacement includes root displacements on the X, Y, and Z axes, and the first displacement can be expressed as: ,in This is the root displacement on the X-axis of the first displacement. This is the root displacement on the Y-axis in the first displacement. Let be the root displacement on the Z-axis in the first displacement. In this case, the first displacement is represented by formula (8).

[0125] (8)

[0126] In formula (8), This represents the root displacement of the HMR coordinate system relative to the first coordinate system. Translation parameters for weak perspective projection in the HMR coordinate system. , The focal length of the HMR virtual camera. The resolution used to annotate the bounding box of the second human body. is the proportional parameter. The meanings of the other parameters in formula (8) are the same as those in formula (6), and are omitted here.

[0127] in, and These are all predefined parameters. For example, , .

[0128] In this case, the projection chain relationship is represented by formula (9).

[0129] (9)

[0130] In formula (9), This represents the coordinates of the second human figure projected onto the 2D panoramic image. This represents the three-dimensional coordinates of the second human body predicted by the HMR regressor in the HMR coordinate system, which is also the first joint feature in step b. The first displacement is calculated using formula (8); Let the second human body be in three-dimensional coordinates in the first coordinate system. This indicates the transformation of three-dimensional coordinates in the HMR coordinate system to three-dimensional coordinates in the first coordinate system. The process. This describes the process of projecting three-dimensional coordinates onto two-dimensional coordinates.

[0131] Step b: Obtain the first joint features of the second human body.

[0132] The first joint feature includes the coordinates of multiple joints of the second human body predicted by the HMR regressor. The first joint feature is a feature in the HMR coordinate system.

[0133] Step c: Using a projection chain relationship, the features of the first joint point are projected onto the two-dimensional panoramic image.

[0134] This involves inputting the features of the first joint point into formula (9) to obtain... .

[0135] Step d: Calculate the two-dimensional reprojection loss of the first joint feature.

[0136] Optionally, step d can be represented by formula (10).

[0137] (10)

[0138] In formula (10), For two-dimensional reprojection loss, Let be the true values ​​of the coordinates of the second human body projected onto the two-dimensional panoramic image. The meanings of the other parameters in formula (10) are the same as those in formula (9), and are omitted here.

[0139] Two-dimensional reprojection loss can accurately reflect the projection deviation of the estimation error of three-dimensional coordinates on the real imaging plane, thus providing a supervision signal that conforms to the actual imaging geometry for three-dimensional human body modeling models.

[0140] The first displacement is calculated using formula (8), and the multiple parameters involved in formula (8) are all parameters in the bounding box geometric features. As can be seen from formula (9), the joint features predicted by the HMR regressor in the HMR coordinate system can be transformed to the first coordinate system based on the first displacement, and then the two-dimensional reprojection loss can be calculated based on formula (10). Therefore, the bounding box geometric features play an important role in transforming the coordinates of multiple joints predicted by the HMR regressor to the first coordinate system and in calculating the projection loss of the joint features in the first coordinate system.

[0141] Step e: Determine the total error loss based on the two-dimensional reprojection loss.

[0142] Optionally, the total error loss is expressed by formula (11).

[0143] (11)

[0144] In formula (11), This represents the total error loss.

[0145] In 3D human body modeling, the first joint features predicted by the HMR regressor can be input into the SMPL model to reconstruct a 3D human body model. Compared with the standard SMPL model, the reconstructed 3D human body model may contain errors. This refers to the error between the 3D human body model reconstructed based on the features of the first joint and the standard SMPL model.

[0146] The first keypoint feature predicted by the HMR regressor is a three-dimensional keypoint. These three-dimensional keypoints are the predicted values ​​of the HMR regressor, and there may be errors between the predicted values ​​and the true values. This is the error between the first joint feature predicted by the HMR regressor and the true value of that first joint feature.

[0147] , , For the weights, where for The weight, for The weight, for The weights. The values ​​of these three weights can be set empirically, and the sum of these three weights is 1. For example, , , .

[0148] Step f: Based on the total error loss, optimize the parameters of the HMR regressor to train the 3D human body modeling model.

[0149] The total error loss is the objective function. When training a 3D human body model, minimizing the total error loss is the goal, and the parameters of the HMR regressor can be gradually optimized. Once the training objective is achieved, training can be stopped, resulting in a successfully trained 3D human body model.

[0150] Structurally, the 3D human body modeling model in this embodiment differs from the conventional HMR architecture in that the convolutional neural network is replaced by a ResNet feature extractor, which is connected to the HMR regressor via a dual-channel pooling attention mechanism. Furthermore, the input to the HMR regressor in this embodiment includes bounding box geometric features.

[0151] During training, the 3D human body modeling model in this embodiment differs from the conventional HMR architecture in that, when calculating the 2D reprojection loss, the key features predicted by the HMR regressor in the HMR coordinate system are first transformed to the first coordinate system based on the bounding box geometric features, and then the key features in the first coordinate system are reprojected in 2D. Since the original camera is more sensitive to slight changes in human posture than the HMR virtual camera when projecting coordinates from 3D space onto a 2D plane, this improves the accuracy of the 3D human body modeling model in recognizing slight posture changes.

[0152] In step 305, human pose estimation is performed based on the three-dimensional human model output by the trained three-dimensional human modeling model.

[0153] Figure 5 This is a structural diagram of the trained 3D human body model. Figure 5 As shown, after a single image is input to the ResNet feature extractor 501, the ResNet feature extractor 501 first extracts a feature vector from the single image. Then, the feature vector is processed by a dual-channel pooling attention mechanism to obtain the image features of the single image. The first fused feature 504, obtained by concatenating the image features with the bounding box geometric features, is input to the Hybrid Multimodal Representation Regressor 502 (i.e., the HMR regressor). The input of the HMR regressor 502 includes the first fused feature, and the output of the HMR regressor 502 includes the predicted keypoint features. The HMR regressor 502 inputs the predicted keypoint features into the skinned multi-person linear model 503 (i.e., the SMPL model), which can then output a three-dimensional human body model.

[0154] Optionally, before the HMR regressor 502 inputs the predicted joint features into the skinned multi-person linear model 503, an adaptive weighting algorithm can be used to optimize each joint feature predicted by the HMR regressor to further improve the reliability of the established three-dimensional human body model.

[0155] There are many implementation methods for adaptive weighting algorithms in related technologies, so they will not be detailed here.

[0156] Figure 6 This is a schematic diagram showing a single image input to a 3D human body model and the 3D human body model output from the 3D human body modeling model. Through... Figure 6 It can be seen that the 3D human body model created by this 3D human body modeling model is quite accurate.

[0157] The following are device embodiments of this application. For details not described in detail in the device embodiments, please refer to the above method embodiments.

[0158] Figure 7 A schematic diagram of a human pose estimation device provided in an exemplary embodiment of this disclosure is shown. See also Figure 7 The human pose estimation device 700 includes: an acquisition module 701, a training module 702, and a human pose estimation module 703.

[0159] The acquisition module 701 is used to acquire the first dataset, which includes multiple images of the first human body.

[0160] The training module 702 is used to train a 3D human body modeling model using the first dataset. The trained 3D human body modeling model is used to generate a 3D human body model based on a single image input into the 3D human body modeling model.

[0161] The human pose estimation module 703 is used to estimate human pose based on the 3D human model output by the trained 3D human modeling model. The 3D human modeling model includes a ResNet feature extractor, a Hybrid Multimodal Representation (HMR) regressor, and a skinned multi-person linear SMPL model connected in sequence. The input of the HMR regressor includes image features of the first image and bounding box geometric features of the second human body. The output of the HMR regressor includes joint features. The first image is the image input to the 3D human modeling model, and the second human body is the human body in the first image. The joint features include multiple joint coordinates. The bounding box geometric features are used to transform the joint features to the first coordinate system to calculate the projection loss of the joint features in the first coordinate system. The first coordinate system is the coordinate system of the original camera, and the original camera is the camera corresponding to the first image.

[0162] Optionally, the images of the first human body in the first dataset include depth RGB images of the first human body at azimuth angles of 0°, 60°, 120°, 180°, 240°, 300°, and 360°, respectively. The ResNet feature extractor includes 6 branches, each branch being used to extract feature vectors from the depth RGB image at a given azimuth angle. The training module 702 is also used to pre-train the 6 branches of the ResNet feature extractor based on the depth RGB images at different azimuth angles in the first dataset. After the 6 branches of the ResNet feature extractor are pre-trained, the parameters of the 6 branches of the ResNet feature extractor are frozen, while the parameters of the HMR regressor are fine-tuned. After fine-tuning the parameters of the HMR regressor, the parameters of the HMR regressor are optimized using the total error loss to train the 3D human body modeling model. The total error loss includes the projection loss of the joint features calculated on the first coordinate system.

[0163] Optionally, the training module 702 is further configured to establish a projection chain relationship based on the first root displacement. The projection chain relationship indicates the process of transforming the 3D coordinates in the HMR coordinate system to the 3D coordinates in the first coordinate system, and projecting the 3D coordinates in the first coordinate system onto the 2D panoramic image. The HMR coordinate system is the coordinate system of the HMR virtual camera, and the HMR virtual camera is the camera corresponding to the bounding box used to annotate the second human body. The first root displacement is the root displacement between the HMR coordinate system and the first coordinate system, and the first root displacement is determined based on the geometric features of the bounding box of the second human body. The module also acquires the first joint feature of the second human body, which includes the coordinates of multiple joints of the second human body predicted by the HMR regressor. The first joint feature is a feature in the HMR coordinate system. The projection chain relationship is used to project the first joint feature onto the 2D panoramic image. The 2D reprojection loss of the first joint feature is calculated. Based on the 2D reprojection loss, the total error loss is determined. Based on the total error loss, the parameters of the HMR regressor are optimized, and the 3D human body modeling model is trained.

[0164] Optionally, the ResNet feature extractor and the HMR regressor are connected via a dual-channel pooling attention mechanism, which includes a global average pooling channel and a global max pooling channel. The global average pooling channel and the global max pooling channel are concatenated through the attention mechanism. The training module 702 is also used to input the feature vectors output from the six branches of the ResNet feature extractor into the dual-channel pooling attention mechanism to obtain the image features of the first image; the image features of the first image are concatenated with the bounding box geometric features to obtain the first fused feature, which is used as input to the HMR regressor.

[0165] It should be noted that the human pose estimation device provided in the above embodiments is only illustrated by the division of the functional modules described above. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the human pose estimation device and the human pose estimation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0166] The module division in this embodiment is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. Furthermore, the functional modules in the various embodiments of this disclosure can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0167] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device (which may be a personal computer, mobile phone, or communication device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. For example... Figure 8 As shown, the computer device 800 includes a processor 801 and a memory 802.

[0169] Processor 801 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 801 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 801 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0170] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 802 is used to store at least one instruction, which is executed by the processor 801 to implement the human pose estimation method provided in the embodiments of this disclosure.

[0171] Those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on the computer device 800, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0172] This disclosure also provides a non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a computer device, the computer device is able to perform the human pose estimation method provided in this disclosure.

[0173] This disclosure also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the human pose estimation method provided in this disclosure.

[0174] The above description is merely an optional embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A method for estimating human pose, characterized in that, The method includes: Obtain a first dataset, which includes multiple images of a first human body; The first dataset is used to train a 3D human body modeling model. The trained 3D human body modeling model is used to generate a 3D human body model based on a single image input into the 3D human body modeling model. Human pose estimation is performed based on the 3D human model output from the trained 3D human modeling model. The 3D human modeling model includes a ResNet feature extractor, a Hybrid Multimodal Representation (HMR) regressor, and a skinned multi-person linear SMPL model connected in sequence. The input of the HMR regressor includes image features of a first image and bounding box geometric features of a second human body. The output of the HMR regressor includes joint features. The first image is the image input to the 3D human modeling model, the second human body is the human body in the first image, and the joint features include multiple joint coordinates. The first coordinate system is the coordinate system of the original camera, and the original camera is the camera corresponding to the first image. The images of the first human body in the first dataset include depth RGB images of the first human body at azimuth angles of 0°, 60°, 120°, 180°, 240°, 300°, and 360°, respectively. The ResNet feature extractor includes six branches, each branch being used to extract a feature vector from the depth RGB image at one of the azimuth angles. The step of training a 3D human body modeling model using the first dataset includes: Based on the depth RGB images at different azimuth angles in the first dataset, the six branches of the ResNet feature extractor are pre-trained. After the six branches of the ResNet feature extractor are pre-trained, the parameters of the six branches of the ResNet feature extractor are frozen, while the parameters of the HMR regressor are fine-tuned. After fine-tuning the parameters of the HMR regressor, the parameters of the HMR regressor are optimized using total error loss to train the three-dimensional human modeling model. The total error loss includes the projection loss of the joint features calculated on the first coordinate system. The optimization of the HMR regressor parameters using total error loss includes: Based on the first displacement, a projection chain relationship is established. The projection chain relationship is used to indicate the process of transforming the three-dimensional coordinates in the HMR coordinate system to the three-dimensional coordinates in the first coordinate system, and projecting the three-dimensional coordinates in the first coordinate system onto the two-dimensional panoramic image. The HMR coordinate system is the coordinate system of the HMR virtual camera. The HMR virtual camera is the camera used to annotate the bounding box corresponding to the second human body. The first displacement is the root displacement between the HMR coordinate system and the first coordinate system. The first displacement is determined based on the geometric features of the bounding box of the second human body. The first joint feature of the second human body is obtained. The first joint feature includes multiple joint coordinates of the second human body predicted by the HMR regressor. The first joint feature is a feature in the HMR coordinate system. Using the aforementioned projection chain relationship, the features of the first joint point are projected onto the two-dimensional panoramic image; Calculate the two-dimensional reprojection loss of the features of the first joint point; Based on the aforementioned two-dimensional reprojection loss, the total error loss is determined; Based on the total error loss, the parameters of the HMR regressor are optimized, and the 3D human body modeling model is trained. The bounding box geometry of the second human body is represented by the following formula: in, The bounding box geometry features, The focal length of the original camera; This indicates the coordinates of the center of the bounding box used to annotate the second human figure in the image coordinate system, which is a two-dimensional coordinate system with the center of the first image as its origin. This indicates the side length of the bounding box used to annotate the second human body. The width of the first image. The height of the first image; The first displacement is obtained using the following formula: in, This represents the root displacement of the HMR coordinate system relative to the first coordinate system. The translation parameters for weak perspective projection in the HMR coordinate system are: , The focal length of the HMR virtual camera. , The resolution used to annotate the bounding box of the second human body. This is a proportional parameter.

2. The method according to claim 1, characterized in that, The ResNet feature extractor and the HMR regressor are connected via a dual-pooling channel attention mechanism, which includes a global average pooling channel and a global max pooling channel. The global average pooling channel and the global max pooling channel are concatenated through an attention mechanism. The method further includes: The feature vectors output from the six branches of the ResNet feature extractor are input into the dual-channel pooling attention mechanism to obtain the image features of the first image. The image features of the first image are concatenated with the geometric features of the bounding box to obtain a first fused feature, which is then input into the HMR regressor.

3. A human posture estimation device, characterized in that, The device includes: The acquisition module is used to acquire a first dataset, which includes multiple images of a first human body. The training module is used to train a 3D human body modeling model using the first dataset. The trained 3D human body modeling model is used to generate a 3D human body model based on a single image input into the 3D human body modeling model. The human pose estimation module is used to estimate human pose based on the 3D human model output by the trained 3D human modeling model. The 3D human modeling model includes a ResNet feature extractor, a Hybrid Multimodal Representation (HMR) regressor, and a skinned multi-person linear SMPL model connected in sequence. The input of the HMR regressor includes image features of a first image and bounding box geometric features of a second human body. The output of the HMR regressor includes joint features. The first image is the image input to the 3D human modeling model, and the second human body is the human body in the first image. The joint features include multiple joint coordinates. The bounding box geometric features are used to transform the joint features to a first coordinate system to calculate the projection loss of the joint features in the first coordinate system. The first coordinate system is the coordinate system of the original camera, and the original camera is the camera corresponding to the first image. The images of the first human body in the first dataset include depth RGB images of the first human body at azimuth angles of 0°, 60°, 120°, 180°, 240°, 300°, and 360°, respectively. The ResNet feature extractor includes six branches, each branch being used to extract a feature vector from the depth RGB image at one of the azimuth angles. The training module is also used to pretrain the six branches of the ResNet feature extractor based on the depth RGB images at different azimuth angles in the first dataset. After the six branches of the ResNet feature extractor are pre-trained, the parameters of the six branches of the ResNet feature extractor are frozen, while the parameters of the HMR regressor are fine-tuned. After fine-tuning the parameters of the HMR regressor, the parameters of the HMR regressor are optimized using total error loss to train the three-dimensional human modeling model. The total error loss includes the projection loss of the joint features calculated on the first coordinate system. The training module is also used to establish a projection chain relationship based on the first root displacement. The projection chain relationship is used to indicate the process of transforming the three-dimensional coordinates in the HMR coordinate system to the three-dimensional coordinates in the first coordinate system and projecting the three-dimensional coordinates in the first coordinate system onto the two-dimensional panoramic image. The HMR coordinate system is the coordinate system of the HMR virtual camera. The HMR virtual camera is the camera used to annotate the bounding box corresponding to the second human body. The first root displacement is the root displacement between the HMR coordinate system and the first coordinate system. The first root displacement is determined based on the geometric features of the bounding box of the second human body. The first joint feature of the second human body is obtained. The first joint feature includes multiple joint coordinates of the second human body predicted by the HMR regressor. The first joint feature is a feature in the HMR coordinate system. Using the aforementioned projection chain relationship, the features of the first joint point are projected onto the two-dimensional panoramic image; Calculate the two-dimensional reprojection loss of the features of the first joint point; Based on the aforementioned two-dimensional reprojection loss, the total error loss is determined; Based on the total error loss, the parameters of the HMR regressor are optimized, and the 3D human body modeling model is trained. The bounding box geometry of the second human body is represented by the following formula: in, The bounding box geometry features, The focal length of the original camera; This indicates the coordinates of the center of the bounding box used to annotate the second human figure in the image coordinate system, which is a two-dimensional coordinate system with the center of the first image as its origin. This indicates the side length of the bounding box used to annotate the second human body. The width of the first image. The height of the first image; The first displacement is obtained using the following formula: in, This represents the root displacement of the HMR coordinate system relative to the first coordinate system. The translation parameters for weak perspective projection in the HMR coordinate system are: , The focal length of the HMR virtual camera. , The resolution used to annotate the bounding box of the second human body. This is a proportional parameter.

4. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement the method of claim 1 or 2.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the method of claim 1 or 2.

6. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method of claim 1 or 2.

Images