Machine learning model training apparatus and method, and three-dimensional pose estimation apparatus

Abstract

The present disclosure relates to the technical field of computers, and relates to a machine learning model training apparatus and method, and a three-dimensional pose estimation apparatus. The training apparatus comprises at least one processor, the at least one processor being configured to: acquire a plurality of two-dimensional images and a machine learning model to be trained, the plurality of two-dimensional images comprising a target, and the machine learning model comprising a first three-dimensional pose estimation module and a second three-dimensional pose estimation module; according to any one two-dimensional image, use the first three-dimensional pose estimation module to estimate a first three-dimensional pose of the target; according to at least two two-dimensional images, use the second three-dimensional pose estimation module to estimate a second three-dimensional pose of the target, the viewing angles of the at least two two-dimensional images being different from each other; and training at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module according to the difference between the first three-dimensional pose and the second three-dimensional pose.

Description

Training apparatus, methods, and 3D pose estimation apparatus for machine learning models

[0001] Cross-reference to related applications

[0002] This application is based on and claims priority to CN application No. 202310691946.6, filed on June 12, 2023, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to the field of computer technology, and in particular to a training device for a machine learning model, a training method for a machine learning model, a device for estimating three-dimensional pose, a method for estimating three-dimensional pose, an electronic device, and a non-volatile computer-readable storage medium. Background Technology

[0004] Monocular viewpoint 3D human pose estimation methods have received widespread attention in the field of computer vision, leading to numerous related applications. For example, machine learning can be used to estimate 3D human pose, resulting in various applications including human-computer interaction, action recognition, and virtual reality.

[0005] In related technologies, with the tremendous success of deep learning models and increasingly complex datasets, deep CNNs (Convolutional Neural Networks) can be applied to estimate 3D human poses in monocular camera settings.

[0006] Summary of the Invention

[0007] According to some embodiments of this disclosure, a training apparatus for a machine learning model is provided, including at least one processor. The at least one processor is configured to: acquire multiple two-dimensional images and a machine learning model to be trained, wherein the multiple two-dimensional images contain a target, and the machine learning model includes a first three-dimensional pose estimation module and a second three-dimensional pose estimation module; estimate the first three-dimensional pose of the target based on any one of the two-dimensional images using the first three-dimensional pose estimation module; estimate the second three-dimensional pose of the target based on at least two two-dimensional images using the second three-dimensional pose estimation module, wherein the viewpoints of the at least two two-dimensional images are different from each other; and train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the first three-dimensional pose and the second three-dimensional pose.

[0008] In some embodiments, the target includes multiple feature points. Estimating the second three-dimensional pose of the target using a second three-dimensional pose estimation module based on at least two two-dimensional images includes: estimating information about the three-dimensional spatial range of each of the multiple feature points of the target based on at least two two-dimensional images; and estimating the second three-dimensional pose based on the information about the three-dimensional spatial range.

[0009] In some embodiments, estimating the information of the three-dimensional spatial extent of each of the plurality of feature segments of the target based on at least two two-dimensional images includes: estimating a two-dimensional feature point heatmap of each of the at least two two-dimensional images; and estimating the information of the three-dimensional spatial extent based on the two-dimensional feature point heatmap.

[0010] In some embodiments, estimating the second three-dimensional pose based on information about the three-dimensional spatial extent includes: estimating a three-dimensional feature point heatmap based on information about the three-dimensional spatial extent; and estimating the second three-dimensional pose based on the three-dimensional feature point heatmap.

[0011] In some embodiments, estimating a second three-dimensional pose based on a three-dimensional feature point heatmap includes: estimating the second three-dimensional pose by performing soft-argmax processing on the three-dimensional feature point heatmap.

[0012] In some embodiments, the three-dimensional spatial extent includes a cubic spatial extent.

[0013] In some embodiments, estimating the three-dimensional spatial extent of each of a plurality of feature points of a target based on at least two two-dimensional images includes: estimating the three-dimensional spatial extent using a 3DPS (3D Pictorial Structures) module.

[0014] In some embodiments, training at least one of a first three-dimensional pose estimation module or a second three-dimensional pose estimation module based on the difference between the first three-dimensional pose and the second three-dimensional pose includes: acquiring a first key point of the first three-dimensional pose and a second key point of the second three-dimensional pose; correcting the viewpoint of the second three-dimensional pose and the viewpoint of the first three-dimensional pose to be consistent based on the first key point and the second key point; and training at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the corrected first three-dimensional pose and the second three-dimensional pose.

[0015] In some embodiments, correcting the viewpoint of the second three-dimensional pose and the viewpoint of the first three-dimensional pose to a consistent viewpoint based on the first keypoint and the second keypoint includes: correcting the viewpoint of the second three-dimensional pose to the viewpoint of the first three-dimensional pose to generate a third three-dimensional pose, so as to train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the third three-dimensional pose and the first three-dimensional pose.

[0016] In some embodiments, obtaining the first key point of the first three-dimensional pose and the second key point of the second three-dimensional pose includes: estimating the first key point and the second key point using a first upscaling network module.

[0017] In some embodiments, the first keypoint is located at the center of the first three-dimensional pose, and the second keypoint is located at the center of the second three-dimensional pose.

[0018] In some embodiments, estimating the first three-dimensional pose of a target using a first three-dimensional pose estimation module based on any two-dimensional image includes: estimating the two-dimensional pose of the target using a first two-dimensional detection module based on any two-dimensional image; and estimating the first three-dimensional pose using a second upscaling network module based on the two-dimensional pose of the target.

[0019] In some embodiments, estimating the two-dimensional pose of a target using a first two-dimensional detection module based on any first two-dimensional image includes: estimating a two-dimensional feature point heatmap of any first two-dimensional image based on any first two-dimensional image; and estimating the two-dimensional pose based on the two-dimensional feature point heatmap.

[0020] In some embodiments, estimating a two-dimensional pose based on a two-dimensional feature point heatmap includes: estimating the two-dimensional pose by performing soft-argmax processing on the two-dimensional feature point heatmap.

[0021] In some embodiments, the second up-dimensional network module includes a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer.

[0022] In some embodiments, training at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the first three-dimensional pose and the second three-dimensional pose includes: determining the MSE (Mean Squared Error) loss and the two-dimensional reprojection loss based on the difference between the first three-dimensional pose and the second three-dimensional pose; and training at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the weighted average of the MSE loss and the two-dimensional reprojection loss until the training termination condition is met.

[0023] In some embodiments, the target includes a human target, and the multiple feature points of the human target include multiple key points.

[0024] In some embodiments, the training apparatus further includes a display device for displaying at least one of a first three-dimensional pose or a second three-dimensional pose.

[0025] According to other embodiments of this disclosure, a three-dimensional pose estimation apparatus is provided, comprising: a training apparatus for a machine learning model in any of the above embodiments, configured to train a first three-dimensional pose estimation module for the machine learning model; and at least one processor configured to: acquire a two-dimensional image containing a target; and estimate the three-dimensional pose of the target based on the two-dimensional image using the trained first three-dimensional pose estimation module.

[0026] In some embodiments, the processor is configured to: obtain the usage requirements of the machine learning model; and convert the machine learning model into a format that matches the usage requirements.

[0027] In some embodiments, the processor is configured to: in response to a user's setting operation, set the deployment environment and model parameters for the machine learning model.

[0028] In some embodiments, the display device is used to display a three-dimensional pose.

[0029] According to further embodiments of this disclosure, a method for training a machine learning model is provided, comprising: estimating a first three-dimensional pose of the target using a first three-dimensional pose estimation module of the machine learning model based on any two-dimensional image containing the target; estimating a second three-dimensional pose of the target using a second three-dimensional pose estimation module of the machine learning model based on at least two two-dimensional images containing the target, wherein the viewpoints of the multiple second two-dimensional images are different from each other; and training at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the first three-dimensional pose and the second three-dimensional pose.

[0030] According to further embodiments of the present disclosure, a method for estimating three-dimensional pose is provided, comprising: training a first three-dimensional pose estimation module of a machine learning model according to the training method of the machine learning model in any of the above embodiments; and estimating the three-dimensional pose of the target using the first three-dimensional pose estimation module based on a two-dimensional image containing the target.

[0031] According to further embodiments of the present disclosure, an electronic device is provided, including: a memory; and a processor coupled to the memory, the processor being configured to execute a training method for a machine learning model or a three-dimensional pose estimation method of any of the above embodiments based on instructions stored in the memory device.

[0032] According to further embodiments of the present disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the training method of the machine learning model or the estimation method of three-dimensional pose in any of the above embodiments.

[0033] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0034] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this application, illustrate exemplary embodiments of this disclosure and are used to explain this disclosure, but do not constitute an undue limitation of this disclosure. In the drawings:

[0035] Figure 1 shows a flowchart of a method for training a machine learning model according to some embodiments of the present disclosure;

[0036] Figure 2 illustrates a schematic diagram of a training method for a machine learning model according to other embodiments of the present disclosure;

[0037] Figure 3 illustrates a schematic diagram of a training method for a machine learning model according to some embodiments of the present disclosure;

[0038] Figure 4 illustrates a schematic diagram of a three-dimensional pose estimation method according to some embodiments of the present disclosure;

[0039] Figures 5a-5e illustrate schematic diagrams of a machine learning model development and deployment platform according to some embodiments of the present disclosure;

[0040] Figure 6a shows a block diagram of a training apparatus for a machine learning model according to some embodiments of the present disclosure;

[0041] Figure 6b shows a block diagram of a three-dimensional pose estimation apparatus according to some embodiments of the present disclosure;

[0042] Figure 7 shows a block diagram of some embodiments of the electronic device of this disclosure;

[0043] Figure 8 shows a block diagram of some other embodiments of the electronic device disclosed herein. Detailed Implementation

[0044] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0045] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0046] The method described above for estimating 3D pose from a monocular camera perspective using neural networks faces some challenges.

[0047] First, most typical neural network models require a large amount of annotated 3D human pose data for training. These methods rely on labeled 3D skeletons as 3D pose annotations for depth prediction, which are collected using label-based motion capture systems. However, label-based motion capture systems are costly. In other words, annotating 3D poses as ground reality using label-based motion capture systems is a costly process.

[0048] Secondly, the mathematical theory of projecting 2D joints into 3D space has been well proven, and neural networks themselves are an approximation of this projection. Therefore, over-reliance on real 3D data may lead to overfitting.

[0049] Furthermore, training from a monocular camera's perspective can lead to depth blur issues. Since multiple different 3D skeletons can be projected onto the same 2D pose in a specific camera viewpoint, estimating 3D pose from a monocular perspective can result in strange 3D poses.

[0050] To address the aforementioned technical problems, this disclosure proposes a self-supervised model. This model requires only unlabeled multi-view data and does not require annotated training data. For example, this disclosure can train a network based on geometric knowledge without requiring labeled 3D poses.

[0051] For example, this disclosure constructs a pipeline following two steps: a 2D skeleton estimator from multiple camera views, and a 2D-to-3D pose upsizing network capable of outputting estimated 3D poses. The skeleton estimator can estimate the 2D coordinates or 2D heatmaps of the feature points that constitute the skeleton model of the target.

[0052] For example, this disclosure also utilizes multi-view consistency constraints to better converge the captured 2D pose to the 3D pose. The multi-view setup also addresses the technical problem of depth blur in cases of incomplete or erroneous 2D detection.

[0053] For example, this disclosure uses the confidence of 2D joints to integrate losses from different viewpoints to mitigate the effects of noise caused by self-occlusion problems.

[0054] To address the aforementioned technical issues, this disclosure proposes a self-supervised training method in a multi-view training setting. By using a self-supervised method instead of explicit 3D ground truth annotations, training can be completed using only the 2D pose estimate generated by a 2D pose estimator as input.

[0055] In this way, not only can the depth blur problem be overcome by setting multiple perspectives, but also the incomplete or erroneous 2D detection can be solved by utilizing information from other perspectives.

[0056] For example, multi-view settings are more advantageous when training models because information from different perspectives and monocular settings have a wider range of applications. This disclosure can apply multi-view settings only during training and monocular settings during testing, thus combining the advantages of both paradigms.

[0057] For example, the structure of this disclosure combines the outputs of two branches of a neural network. The first branch takes a single image (from a camera viewpoint) as input and generates a 3D pose in 3D space; the second branch takes multiple images from different camera views as input and outputs estimated 3D poses. The method provided in this disclosure allows all estimated 3D poses to be projected from both branches onto any camera viewpoint.

[0058] In some embodiments, the method provided in this disclosure may include two stages. The first stage uses a 2D pose estimator to estimate the 2D human pose; the second stage upscales these 2D estimates to 3D space; and combines the camera's geometric information, the 3D pose from the second branch, and the rotation matrix from the viewpoint of the first branch to obtain the desired 3D pose in the first camera coordinate system through rotation and other processing.

[0059] In other words, all 3D poses in the real-world coordinate system should be completely consistent, so each acquired pose can be projected back into its own 3D camera coordinate system. Furthermore, by reprojecting the 3D poses onto each 2D camera viewpoint using the camera matrix, a reprojection loss can be defined for each 3D-to-2D projection.

[0060] In some embodiments, this disclosure proposes a two-branch self-supervised method in a multi-view training setting to train 2D to 3D upscaling networks without requiring 3D data annotation. The technical solution of this disclosure relies solely on geometric knowledge to construct the supervision signal, thereby achieving better generalization ability.

[0061] In some embodiments, this disclosure proposes a recurrent viewpoint training scheme that is effective in utilizing multi-viewpoint consistency information and in constraining 3D pose estimation during training. This overcomes the depth blur problem and allows the use of information from other viewpoints to handle incomplete or erroneous 2D detection cases. Furthermore, this disclosure utilizes joint 2D confidence from different camera viewpoints to mitigate the self-occlusion problem.

[0062] In some embodiments, this disclosure proposes a volumetric 3DPS network based on a voxel method and 3DPS. This network is capable of exploring a sufficient state space in which volumetric cubes are generated by mesh sampling; the cubes are then fed into a voxel-based neural network to obtain 3D poses.

[0063] The inventors of this disclosure have discovered the following problem in the aforementioned related technologies: machine learning models require a large number of labeled data samples, resulting in high costs for pose estimation. In view of this, this disclosure proposes a training technique for machine learning models. For example, the technical solution of this disclosure can be implemented through the following embodiments.

[0064] Figure 1 shows a flowchart of a training method for a machine learning model according to some embodiments of the present disclosure.

[0065] As shown in Figure 1, in step 110, multiple two-dimensional images and a machine learning model to be trained are acquired. The multiple two-dimensional images contain a target. The machine learning model includes a first three-dimensional pose estimation module and a second three-dimensional pose estimation module. For example, the target may include a human target, and multiple feature points of the human target may include multiple joint points.

[0066] In some embodiments, the machine learning model includes a first 3D pose estimation module and a second 3D pose estimation module. The first 3D pose estimation module includes a first 2D detection module and a second upscaling network module; the second 3D pose estimation module includes a second 2D detection module and a 3D PS module. The first 2D detection module and the second 2D detection module may be the same or different.

[0067] In some embodiments, with four camera viewpoints, there are four synchronized camera viewpoints and camera projection matrices, and all cameras capture two-dimensional images of the same target and the same scene.

[0068] For example, multiple two-dimensional images processed by a machine learning model are acquired based on the perspective and projection matrix of multiple synchronized shooting devices to ensure that the multiple two-dimensional images remain synchronized in time, thereby improving the training effect and processing accuracy of the machine learning model.

[0069] In step 120, the first three-dimensional pose of the target is estimated using a first three-dimensional pose estimation module based on any given two-dimensional image. For example, the two-dimensional image is acquired from a first shooting angle using a monocular camera.

[0070] In this way, the training of the first 3D pose estimation module can be completed without labeling the training samples. Once trained, this first 3D pose estimation module can accurately estimate the 3D pose of a target in a monocular camera scenario without the need for multiple cameras, thus reducing costs.

[0071] In some embodiments, the first three-dimensional pose may include at least one of a three-dimensional heatmap or three-dimensional coordinates.

[0072] For example, when using a pre-trained 3D pose estimation module, the system can output the first 3D pose in 3D coordinates to the user, thus providing more accurate information. The user can then perform subsequent operations such as behavior recognition and action alerts based on these 3D coordinates.

[0073] For example, during the training of the first 3D pose estimation module, the first 3D pose can be output in the form of a 3D heatmap. The heatmap can represent the range of feature points rather than a fixed point, providing the system with a higher fault tolerance and thus improving the training effect of the first 3D pose estimation module.

[0074] In some embodiments, the two-dimensional pose of the target is estimated using a first two-dimensional detection module based on any two-dimensional image.

[0075] For example, a 2D pose estimator can be used to predict 2D pose. For each frame of a 2D image, the 2D image can be cropped based on the bounding box containing the target detected by the 2D detector.

[0076] For example, the cropped 2D image from the perspective of the first camera out of four cameras is represented by I, and the cropped 2D images from the perspectives of the other three cameras are represented by I. This means that each cropped 2D image is input into a 2D pose estimator to estimate the 2D pose X of the target.

[0077] For example, the backbone of a two-dimensional pose estimator is denoted by f, and the weight parameter is θ. A two-dimensional pose estimator can consist of two parts: a global network for coarsely predicting the two-dimensional pose; and a fine network for refining the joints.

[0078] In some embodiments, a two-dimensional feature point heatmap of any first two-dimensional image is estimated based on any first two-dimensional image; and a two-dimensional pose is estimated based on the two-dimensional feature point heatmap. For example, the two-dimensional pose is estimated by performing soft-argmax processing on the two-dimensional feature point heatmap.

[0079] For example, a heatmap H of I can be estimated using a two-dimensional attitude estimator: H = f(I; θ)

[0080] For example, the two-dimensional pose X of a target can be estimated using H. The argmax algorithm can be used to estimate the two-dimensional pose X. The argmax algorithm obtains the joint with the highest probability by calculating the maximum point of each heatmap.

[0081] However, the argmax algorithm may sever the gradient flow, making it impossible to train a 2D pose estimator. To address this issue, soft-argmax can be applied to the heatmap H as an alternative to argmax.

[0082] H k Let Ω be the heatmap of the k-th joint of the target captured by the first camera, and let Ω be the domain of the heatmap. Soft-argmax utilizes the probability G corresponding to position i. k (i) Weight position i to obtain the weighted coordinates of position i; add all the weighted coordinates together to obtain the two-dimensional coordinates of each joint as the two-dimensional pose of the target.

[0083] In the above embodiments, soft-argmax processing can ensure that the gradient flow is transferred from the output 3D pose to the input 2D image, so that the machine learning model does not cut off the gradient flow, thereby improving the accuracy of pose estimation.

[0084] In some embodiments, the first three-dimensional pose is estimated using a second upscaling network module based on the target's two-dimensional pose. For example, the N detected 2D joints are represented as... Predicting 3D pose using a 3D upscaling network.

[0085] In some embodiments, the second up-dimensional network module includes a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer.

[0086] For example, the updimensional network w vThe goal is to estimate the joint positions of a target in 3D space given only 2D input. The upscaling network can include batch normalization layers, dropout layers, rectified linear units, and residual connection layers via residual connections. The input layer of the upscaling network can take N (e.g., a positive integer like 17) coordinates of human joints as input and apply them to a fully connected layer with 1024 output channels; then, it is processed using a residual connection layer consisting of four modules via residual connections, each module consisting of two fully connected layers; then, it is processed using batch normalization layers, rectified linear units, and dropout layers; finally, the final features output by one residual module are input into a linear layer to obtain the 3D pose Y: Y = w v (X)

[0087] In the above embodiments, a 2D pose estimation network is applied to predict the 2D pose and heatmap of multiple input frame images. Then, the confidence of all 2D feature points is boosted to the confidence of 3D feature points to integrate the losses from different viewpoints. This can mitigate the impact of noise caused by self-occlusion, thereby improving the accuracy of pose estimation.

[0088] In step 130, the second three-dimensional pose of the target is estimated using the second three-dimensional pose estimation module based on at least two two-dimensional images, wherein the viewpoints of the at least two two-dimensional images are different from each other.

[0089] For example, the second three-dimensional pose may include at least one of the three-dimensional heatmap of the target or the three-dimensional coordinates of each feature point of the target.

[0090] In some embodiments, the at least two two-dimensional images can be acquired by a monocular camera from different shooting angles, or by a multi-camera system from different shooting angles. For example, the two two-dimensional images can be acquired by a monocular camera from a second angle and a third angle, respectively, or by one of a binocular camera from a second angle and the other from a third angle. The second angle and the third angle are different.

[0091] In some embodiments, the target includes multiple feature points. Based on at least two two-dimensional images, information about the three-dimensional spatial extent of each of the multiple feature points of the target is estimated; based on the information about the three-dimensional spatial extent, a second three-dimensional pose is estimated. For example, the three-dimensional spatial extent includes a cubic spatial extent, a spherical spatial extent, etc.

[0092] In some embodiments, the three-dimensional spatial range corresponding to a feature point can be a spatial range centered on that feature point. For example, a 3DPS network module can be used to estimate a three-dimensional heatmap of the location of the feature point, and the three-dimensional spatial range corresponding to the feature point can be determined based on the three-dimensional heatmap.

[0093] For example, the size of the three-dimensional spatial range can be determined by a preset threshold. A heat threshold can be used to define the three-dimensional spatial range of a feature point as the area in the three-dimensional heatmap where the heat intensity exceeds the threshold.

[0094] In some embodiments, a two-dimensional feature point heatmap is estimated for each of at least two two-dimensional images; based on the two-dimensional feature point heatmap, information about the three-dimensional spatial extent is estimated. For example, the 3DPS module is used to estimate the information about the three-dimensional spatial extent.

[0095] For example, receiving input two-dimensional images from three different camera perspectives. It also outputs a two-dimensional heatmap for each viewpoint's two-dimensional image.

[0096] In some embodiments, the 3DPS module is a multi-view 3D pose prediction method that can explore a sufficient state space and generate 3D human body part candidates through mesh sampling; then, predict 2D pose through a 2D detector; and generate 3D pose using 2D prior and maximum likelihood estimation.

[0097] In some embodiments, to improve the accuracy and robustness of 3DPS, a volumetric 3DPS network can be obtained by combining the volumetric algorithm and 3DPS to estimate the 3D pose.

[0098] For example, a volumetric 3DPS network is used to receive the input two-dimensional heatmap. After processing, the three-dimensional pose is output.

[0099] For example, instead of sampling the 3D joints themselves, a 3DPS module can be used to sample the 3D cubes surrounding the joints of the human body. The 3DPS method can then be applied to generate a 2D heatmap of joint k at viewpoint c. After processing, the volume cube corresponding to joint k under viewpoint c is obtained as the cube's spatial range:

[0100] It is a volume cube generated based on the heatmap of the k-th joint from the viewpoint of the c-th camera, where T represents the processing of the 3DPS method. For example, the size of the volume cube can be 32×32×32.

[0101] The volume cubes from each viewpoint can be summed to obtain the sum of the target's volume cubes.

[0102] This represents the sum of the volume cubes of joint k under multiple viewpoints c.

[0103] In some embodiments, a three-dimensional feature point heatmap is estimated based on information about the three-dimensional spatial extent; and a second three-dimensional pose is estimated based on the three-dimensional feature point heatmap.

[0104] For example, the sum of the volumes of cubes is input into a learnable volume convolutional neural network p. q The architecture of a volumetric convolutional neural network can be similar to that of a voxel-to-voxel network. This can yield a 3D feature point heatmap H. 3D H 3D =p q (V sum )

[0105] V sum The sum of the volumes of the cubes at each joint k The concatenation result is given by q, where q represents the weight from voxel to voxel network.

[0106] In some embodiments, the second three-dimensional pose is estimated by performing soft-argmax processing on the three-dimensional feature point heatmap.

[0107] For example, applying soft-argmax to heatmap H 3D To obtain the three-dimensional pose

[0108] This represents the heatmap of the estimated target at the k-th joint in 3D space, where Ω denotes the domain; thus, the position of the k-th 3D joint can be obtained. As a three-dimensional pose:

[0109] In the above embodiments, a volumetric algorithm (i.e., estimating the 3D spatial range corresponding to feature points) and 3DPS are combined to sample the 3D spatial range around the feature points of the target, and the 3D spatial range is sent to a voxel-based neural network to obtain the 3D pose. This improves the accuracy and robustness of 3DPS, thereby improving the accuracy of pose estimation.

[0110] In step 140, at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained based on the difference between the first three-dimensional pose and the second three-dimensional pose.

[0111] In the above embodiments, the machine learning model of this disclosure is trained in a self-supervised manner in a multi-view setting, so that the machine learning model benefits from information from different perspectives, thereby improving the accuracy of pose estimation.

[0112] However, since the training data lacks annotations of the true 3D pose values, the 3D poses generated from the two branches are unlikely to be located at the same absolute 3D position, potentially causing training to fail to converge. To address this technical problem, a pre-training scheme can be applied to help locate the 3D pose, ensuring that the 3D poses generated from the two branches are located at the same absolute 3D position. For example, the above technical solution can be implemented through the following embodiment.

[0113] In some embodiments, a first key point of the first three-dimensional pose and a second key point of the second three-dimensional pose are obtained; based on the first key point and the second key point, the viewpoint of the second three-dimensional pose and the viewpoint of the first three-dimensional pose are corrected to be consistent; based on the difference between the corrected first three-dimensional pose and the second three-dimensional pose, at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained.

[0114] In some embodiments, a first keypoint and a second keypoint are estimated using a first upscaling network module. For example, the first keypoint is located at the center of a first 3D pose, and the second keypoint is located at the center of a second 3D pose. For example, the pelvis of a human body can be used as the keypoint at the center. For example, the locations of the first and second keypoints can be estimated based on various feature points.

[0115] For example, a first upscaling network, different from the second upscaling network module, can be used to predict the center position of a 3D pose. The first upscaling network module and the second upscaling network can have the same network structure but different weight parameters. The first upscaling network module may include u γ and Two parts, u γ Used to predict the first key point Y from the first camera's perspective. pelvis , Used to predict the second keypoint from randomly selected viewpoints from the second, third, and fourth cameras. Y pelvis =u γ (X)

[0116] The first is the predicted 2D pose from a randomly selected viewpoint. Then, the predicted keypoints can be used to locate the 3D pose, and during training, the 3D pose is moved to the predicted position to ensure that the viewpoint of the second 3D pose is corrected to be consistent with the viewpoint of the first 3D pose.

[0117] In some embodiments, the viewpoint of the second three-dimensional pose is corrected to the viewpoint of the first three-dimensional pose to generate a third three-dimensional pose, so as to train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the third three-dimensional pose and the first three-dimensional pose.

[0118] For example, the correspondence between feature points in the second and first 3D poses can be determined; based on the first and second keypoints, and the positions of corresponding feature points on the edges of the second and first 3D poses, single-line alignment can be performed. This ensures that the viewpoints of the second and first 3D poses are consistent, thereby improving the accuracy of pose estimation.

[0119] In some embodiments, the MSE loss and the two-dimensional reprojection loss are determined based on the difference between the first three-dimensional pose and the second three-dimensional pose; at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained based on the weighted average of the MSE loss and the two-dimensional reprojection loss until the training termination condition is met.

[0120] For example, two different loss functions, the 3D MSE loss and the 2D reprojection loss, can be applied to train a machine learning model. The reprojection loss can further constrain the consistency across multiple views, thereby improving the performance of the machine learning model.

[0121] This is because, without depth information, a 2D detector corresponds to multiple possibilities of 3D pose in different directions. By reprojecting 2D, information from other perspectives can be used to help constrain it to the correct orientation, thereby improving the performance of the machine learning model.

[0122] For example, the MSE loss can be:

[0123] Y k Let k be the first three-dimensional pose of feature point k. The second three-dimensional pose of feature point k.

[0124] For example, the 2D reprojection loss is: P c =[p1 p2 p3] T .

[0125] p1 p2 p3 represent the coordinate transformation information required for the coordinate transformation.

[0126] In some embodiments, the total loss used to train the machine learning model is:

[0127] α is a weighting coefficient that is set as needed to adjust the importance of the two losses during training.

[0128] Figure 2 illustrates a schematic diagram of a training method for a machine learning model according to other embodiments of the present disclosure.

[0129] As shown in Figure 2, in step 210, the first three-dimensional pose estimation module of the machine learning model is trained according to the training method in any of the above embodiments.

[0130] In step 220, a two-dimensional image containing the target is acquired; based on the two-dimensional image, the three-dimensional pose of the target is estimated using the first three-dimensional pose estimation module that has completed training.

[0131] In some embodiments, the usage requirements of the machine learning model are obtained; the machine learning model is converted into a format that matches the usage requirements.

[0132] In some embodiments, in response to a user's settings operation, the deployment environment and model parameters of the machine learning model are configured.

[0133] In some embodiments, a three-dimensional pose is displayed.

[0134] In some embodiments, the target includes multiple feature points, and information about the three-dimensional spatial range of each of the multiple feature points of the target is estimated based on at least two two-dimensional images; a second three-dimensional pose is estimated based on the information about the three-dimensional spatial range.

[0135] In some embodiments, a two-dimensional feature point heatmap of each of at least two two-dimensional images is estimated; based on the two-dimensional feature point heatmap, information about the three-dimensional spatial extent is estimated.

[0136] In some embodiments, a three-dimensional feature point heatmap is estimated based on information about the three-dimensional spatial extent; and a second three-dimensional pose is estimated based on the three-dimensional feature point heatmap.

[0137] In some embodiments, the second three-dimensional pose is estimated by performing soft-argmax processing on the three-dimensional feature point heatmap.

[0138] In some embodiments, the three-dimensional spatial extent includes a cubic spatial extent.

[0139] In some embodiments, the 3DPS module is used to estimate information about the three-dimensional spatial extent.

[0140] In some embodiments, a first key point of the first three-dimensional pose and a second key point of the second three-dimensional pose are obtained; based on the first key point and the second key point, the viewpoint of the second three-dimensional pose and the viewpoint of the first three-dimensional pose are corrected to be consistent; based on the difference between the corrected first three-dimensional pose and the second three-dimensional pose, at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained.

[0141] In some embodiments, the viewpoint of the second three-dimensional pose is corrected to the viewpoint of the first three-dimensional pose to generate a third three-dimensional pose, so as to train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the third three-dimensional pose and the first three-dimensional pose.

[0142] In some embodiments, a first up-dimensional network module is used to estimate a first keypoint and a second keypoint.

[0143] In some embodiments, the first keypoint is located at the center of the first three-dimensional pose, and the second keypoint is located at the center of the second three-dimensional pose.

[0144] In some embodiments, the two-dimensional pose of the target is estimated using a first two-dimensional detection module based on any two-dimensional image; and the first three-dimensional pose is estimated using a second upscaling network module based on the two-dimensional pose of the target.

[0145] In some embodiments, a two-dimensional feature point heatmap of any first two-dimensional image is estimated based on any first two-dimensional image; and a two-dimensional pose is estimated based on the two-dimensional feature point heatmap.

[0146] In some embodiments, the two-dimensional pose is estimated by performing soft-argmax processing on the two-dimensional feature point heatmap.

[0147] In some embodiments, the second up-dimensional network module includes a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer.

[0148] In some embodiments, the MSE loss and the two-dimensional reprojection loss are determined based on the difference between the first three-dimensional pose and the second three-dimensional pose; at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained based on the weighted average of the MSE loss and the two-dimensional reprojection loss until the training termination condition is met.

[0149] In some embodiments, the target includes a human target, and the multiple feature points of the human target include multiple key points.

[0150] In some embodiments, at least one of a first three-dimensional pose or a second three-dimensional pose is displayed.

[0151] Figure 3 illustrates a schematic diagram of a training method for a machine learning model according to some embodiments of the present disclosure.

[0152] As shown in Figure 3, the machine learning model consists of two branches. The first branch takes a single 2D image from the first camera viewpoint as input and generates a first 3D pose in 3D space. The second branch takes three 2D images from the other three camera views as input and outputs estimated second 3D poses. The second 3D pose can be rotated to the first camera viewpoint to achieve a multi-view consistency loss. This ensures that the 3D poses estimated from different viewpoints have the same skeleton before geometric transformation. The predicted 3D poses can be reprojected to each camera viewpoint to obtain additional reprojection error.

[0153] In some embodiments, the machine learning model includes the two-dimensional detector, the upscaling network, and the volumetric 3DPS network shown in Figure 3. The two-dimensional detector includes a first two-dimensional detection module and a second two-dimensional detection module in any embodiment; the upscaling network includes a second upscaling network module in any embodiment; and the volumetric 3DPS network includes the 3DPS module in any embodiment.

[0154] For example, the first three-dimensional pose estimation module includes a two-dimensional detector and an up-dimensional network as shown in Figure 3; the second three-dimensional pose estimation module includes a two-dimensional detector and a volumetric 3DPS network as shown in Figure 3.

[0155] In some embodiments, a two-dimensional detector is applied to predict the two-dimensional pose and two-dimensional heatmap from four input two-dimensional image frames; then, an upscaling network is used to upscale the confidence of each two-dimensional feature point to three dimensions.

[0156] Figure 3 illustrates the network structure for four camera views. There are four synchronized camera views and camera projection matrices, with all cameras capturing the same target and the same scene. First, a 2D detector is used to obtain the 2D pose; for each frame of the 2D image, the bounding box of the target detected by the 2D detector is used to crop the 2D image.

[0157] For example, the cropped 2D image from the perspective of the first camera out of four cameras is represented by I, and the cropped 2D images from the perspectives of the other three cameras are represented by I. The process involves inputting each cropped 2D image into a 2D pose estimator to estimate the target's 2D pose X. For example, the backbone of the 2D pose estimator is represented by f, and the weight parameter is θ. The 2D pose estimator can consist of two parts: a global network for coarsely predicting the 2D pose, and a fine network for refining the joints.

[0158] As shown in Figure 3, the training process has two branches. In the first branch, ... This is represented by N detected 2D joints; then, a 3D upscaling network is used to predict the 3D pose. In another branch, input 2D images from three camera views are received, and a 2D heatmap is output for each view. Then, using a volumetric 3DPS network, the heatmap is received as input and the 3D pose is output. Heatmaps can be estimated using a two-dimensional detector: H = f(I; θ),

[0159] The 2D pose X for the first branch is estimated using H. The 2D pose X can be estimated using the argmax algorithm. The argmax algorithm is able to obtain the joint with the highest probability by calculating the maximum point of each heatmap.

[0160] However, the argmax algorithm may sever the gradient flow, making it impossible to train a 2D pose estimator. To address this issue, soft-argmax can be applied to the heatmap H as an alternative to argmax.

[0161] In some embodiments, the three-dimensional pose can be estimated with zero as the center, Y and The value is relative to the three-dimensional position of the fixed root joint. The predicted three-dimensional pose is in its own pose coordinate system during training, and then can be... Rotating to the first camera coordinates yields the 3D loss; reprojecting the global 3D pose into the second external space yields the pose reprojection loss for each viewpoint.

[0162] For example, lifting networks v The goal is to estimate the joint positions of a target in 3D space given only 2D input. The upscaling network can include batch normalization layers, dropout layers, rectified linear units, and residual connection layers via residual connections. The input layer of the upscaling network can take N (e.g., a positive integer like 17) coordinates of human joints as input and apply them to a fully connected layer with 1024 output channels; then, it is processed using a residual connection layer consisting of four modules via residual connections, each module consisting of two fully connected layers; then, it is processed using batch normalization layers, rectified linear units, and dropout layers; finally, the final features output by one residual module are input into a linear layer to obtain the 3D pose Y: Y = w v (X)

[0163] The 3DPS module is a multi-view 3D pose prediction method that can explore a sufficient state space and generate 3D human body part candidates through mesh sampling; then, it predicts 2D pose through a 2D detector; and finally, it generates 3D pose using 2D priors and maximum likelihood estimation.

[0164] To improve the accuracy and robustness of 3DPS, a volumetric 3DPS network can be obtained by combining volumetric algorithms and 3DPS to estimate 3D pose. Instead of sampling the 3D joints themselves, the 3DPS module can sample the 3D cubes around the joints of the human body. The 3DPS method is then applied to generate a 2D heatmap of joint k at viewpoint c. After processing, the volume cube corresponding to joint k under viewpoint c is obtained as the cube's spatial range:

[0165] It is a volume cube generated based on the heatmap of the k-th joint from the viewpoint of the c-th camera, where T represents the processing of the 3DPS method. For example, the size of the volume cube can be 32×32×32.

[0166] The volume cubes from each viewpoint can be summed to obtain the sum of the target's volume cubes.

[0167] This represents the sum of the volume cubes of joint k under multiple viewpoints c.

[0168] The sum of the volume cubes is input into a learnable volume convolutional neural network p. q The architecture of a volumetric convolutional neural network can be similar to that of a voxel-to-voxel network. This can yield a 3D feature point heatmap H. 3D H 3D =p q (V sum )

[0169] V sum The sum of the volumes of the cubes at each joint k The concatenation result is given by q, where q represents the weight from voxel to voxel network.

[0170] Apply soft-argmax to heatmap H 3D To obtain the three-dimensional pose

[0171] This represents the heatmap of the estimated target at the k-th joint in 3D space, where Ω denotes the domain; thus, the position of the k-th 3D joint can be obtained. As a three-dimensional pose:

[0172] For example, using the L2 norm between 3D poses predicted from different viewpoints. It ensures consistency across multiple viewpoints. If the prediction is accurate, the loss should be zero. This is because the same target captured from different camera perspectives should have the same absolute 3D pose in 3D space.

[0173] However, this solution performs poorly for scenarios with more than two viewpoints. During training, the L2 norm loss of the viewpoints of 2D image 1 and 2D image 2 conflicts with the L2 norm loss of the viewpoints of 2D image 2 and 2D image 3. This is because, without depth information from the camera's viewpoint, the upscaling network learns a 3D pose applicable only to one viewpoint.

[0174] To address the aforementioned technical issues, 3D poses generated from different camera perspectives can be transformed into the same 3D pose. For example, a recurrent viewpoint training scheme can be employed. The machine learning model in Figure 3 consists of two branches: the first branch represents a single camera perspective followed by an upscaling network; the second branch represents three camera perspectives followed by a volumetric 3DPS network. The input 2D images for these four perspectives can be randomly swapped, allowing training of all possible combinations of 3D perspectives to solve the multi-view consistency problem.

[0175] For example, after one round of training using the training method in any of the above embodiments, one two-dimensional image can be selected from at least two two-dimensional images in the previous round of training as the first two-dimensional image, the remaining one of the at least two two-dimensional images can be determined as the second two-dimensional image, and any two-dimensional image in the previous round of training can be determined as the second two-dimensional image; based on the first two-dimensional image, the first three-dimensional pose estimation module is used to estimate the first three-dimensional pose of the target in this round of training; based on the second two-dimensional image, the second three-dimensional pose estimation module is used to estimate the second three-dimensional pose of the target in this round of training; based on the difference between the first three-dimensional pose and the second three-dimensional pose in this round of training, at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained.

[0176] In the above embodiments, the machine learning model of this disclosure is trained in a self-supervised manner in a multi-view setting, so that the machine learning model benefits from information from different perspectives, thereby improving the accuracy of pose estimation.

[0177] However, since the training data lacks annotations of the true 3D pose values, the 3D poses generated from the two branches are unlikely to be located at the same absolute 3D position, potentially causing training to fail to converge. To address this technical problem, a pre-training scheme can be applied to help locate the 3D pose, ensuring that the 3D poses generated from the two branches are located at the same absolute 3D position.

[0178] For example, a first upscaling network, different from the second upscaling network module, can be used to predict the center position of a 3D pose. The first upscaling network module and the second upscaling network can have the same network structure but different weight parameters. The first upscaling network module may include u γ and Two parts, u γ Used to predict the first key point Y from the first camera's perspective. pelvis , Used to predict the second keypoint from randomly selected viewpoints from the second, third, and fourth cameras. Y pelvis =u γ (X)

[0179] Then, the predicted keypoints can be used to locate the 3D pose, and the 3D pose can be moved to the predicted position during training to ensure that the viewpoint of the second 3D pose is corrected to be consistent with the viewpoint of the first 3D pose.

[0180] For example, the double arrows between the first and third 3D poses in Figure 3 can represent training a machine learning model by comparing the differences between the first and third 3D poses.

[0181] For example, soft-argmax can be used to process the 3D heatmap generated by the upscaling network to obtain the first 3D pose. Soft-argmax processing ensures that the gradient flow is transmitted from the output 3D pose to the input 2D image, so that the machine learning model does not cut off the gradient flow.

[0182] Two different loss functions, the 3D MSE loss and the 2D reprojection loss, can be applied to train the machine learning model. The reprojection loss can further constrain the consistency across multiple views, thereby improving the performance of the machine learning model.

[0183] This is because, without depth information, a 2D detector corresponds to multiple possibilities of 3D pose in different directions. 2D reprojection can help constrain it to the correct orientation by utilizing information from other perspectives, thereby improving the performance of machine learning models.

[0184] For example, the MSE loss can be:

[0185] Y k Let k be the first three-dimensional pose of feature point k. The second three-dimensional pose of feature point k.

[0186] For example, the 2D reprojection loss is: P c =[p1 p2 p3] T .

[0187] p1 p2 p3 represent the coordinate transformation information required for the coordinate transformation.

[0188] In some embodiments, the total loss used to train the machine learning model is:

[0189] α is a weighting coefficient that is set as needed to adjust the importance of the two losses during training.

[0190] For example, multi-view 3D human pose estimation methods can use input 2D images from multiple different viewpoints, thus performing better than using a single-eye camera viewpoint.

[0191] In some embodiments, during the training phase, a recurrent viewpoint-assisted refinement of the single-view process and a voxel V2V 3DPS (i.e., volumetric 3DPS) neural network can be utilized. The volumetric 3DPS network includes a 3DPS module and a V2V network for estimating the cube in which the feature points are located.

[0192] For example, a multi-view setup can be used only during the training phase, while a single-view setup is sufficient during the testing phase. This eliminates the need for 3D data annotation to train a 3D pose estimation model.

[0193] For example, reprojection loss helps constrain multi-view information during training. By using a voxel-based V2V 3DPS neural network and a training process based on recurrent perspectives, technical problems caused by data annotation can be avoided, enabling an unlabeled training process, thereby reducing training costs and barriers to entry.

[0194] Figure 4 shows a schematic diagram of a three-dimensional pose estimation method according to some embodiments of the present disclosure.

[0195] As shown in Figure 4, the machine learning model inputs each frame of 2D image into a bounding box detector to obtain cropped 2D images, which are then fed into the backbone network of a 2D pose estimator. The estimated 2D pose is then input into an upscaling network to obtain the estimated 3D pose. For example, the multi-view setting is only used during the training phase, while the monocular camera setting is followed during the testing phase.

[0196] Figures 5a-5e illustrate schematic diagrams of a machine learning model development and deployment platform according to some embodiments of the present disclosure.

[0197] As shown in Figure 5a, multi-view pedestrian data images (i.e., 2D images) are uploaded to the data upload page, and the temporal correspondence of each pedestrian data image is established according to the corresponding naming rules. During data upload, apart from the temporal information transmitted by the camera itself, no manual data annotation is required to support the subsequent training process and test deployment.

[0198] For example, the data upload page includes a task details area and a data file list area. The task details area can include information related to this training task, and the data file list area can include information related to the uploaded data.

[0199] As shown in Figure 5b, on the model building page, users can upload and write the code for the machine learning model to be trained using a notebook. The uploaded code can include the settings and related procedures for training and testing the model.

[0200] For example, the model upload and writing areas on the model building page include information about the notebook, development environment, etc.

[0201] As shown in Figure 5c, on the model training page, the corresponding parameters of the model are designed and adjusted accordingly. During model training, a multi-branch perspective is needed to assist the training process, thereby achieving better training results, while eliminating the need for training labels.

[0202] For example, the model configuration area on the model training page includes functions for configuring training tasks and lithium oxide resources.

[0203] As shown in Figure 5d, on the model conversion page, users can convert trained models to facilitate subsequent deployment. For example, machine learning models can be converted from PyTorch or similar architectures to models using ONNX. During model deployment, model conversion and accuracy correction can meet the diverse needs of different customers.

[0204] For example, the conversion configuration area on the model conversion page includes information about the machine learning model and conversion configuration functions.

[0205] As shown in Figure 5e, on the model deployment page, select the trained model and set the relevant deployment environment and parameters to ensure that the model can run correctly when it is called.

[0206] For example, machine learning models can use the HTTP (Hypertext Transfer Protocol) protocol. The test input to the model can be a single, single-viewpoint image of a pedestrian. The trained model can then identify the 3D pose of pedestrians within the current image, providing a basis for subsequent action recognition or behavior analysis.

[0207] During the deployment phase, this model only needs to use the single-process weight model in the test file to adapt to the needs of the real environment and reduce the hardware costs and conditions required for operation.

[0208] For example, the model deployment area on the model deployment page includes information about the model and information about the services to be processed.

[0209] Figure 6a shows a block diagram of a training apparatus for a machine learning model according to some embodiments of the present disclosure.

[0210] As shown in Figure 6a, the training device 6a for the machine learning model includes at least one processor 61a, which is configured to: acquire multiple two-dimensional images and a machine learning model to be trained, wherein the multiple two-dimensional images contain a target, and the machine learning model includes a first three-dimensional pose estimation module and a second three-dimensional pose estimation module; estimate the first three-dimensional pose of the target using the first three-dimensional pose estimation module based on any one of the two-dimensional images; estimate the second three-dimensional pose of the target using the second three-dimensional pose estimation module based on at least two two-dimensional images, wherein the viewpoints of the at least two two-dimensional images are different from each other; and train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the first three-dimensional pose and the second three-dimensional pose.

[0211] In some embodiments, the target includes multiple feature points, and information about the three-dimensional spatial range of each of the multiple feature points of the target is estimated based on at least two two-dimensional images; a second three-dimensional pose is estimated based on the information about the three-dimensional spatial range.

[0212] In some embodiments, a two-dimensional feature point heatmap of each of at least two two-dimensional images is estimated; based on the two-dimensional feature point heatmap, information about the three-dimensional spatial extent is estimated.

[0213] In some embodiments, a three-dimensional feature point heatmap is estimated based on information about the three-dimensional spatial extent; and a second three-dimensional pose is estimated based on the three-dimensional feature point heatmap.

[0214] In some embodiments, the second three-dimensional pose is estimated by performing soft-argmax processing on the three-dimensional feature point heatmap.

[0215] In some embodiments, the three-dimensional spatial extent includes a cubic spatial extent.

[0216] In some embodiments, the 3DPS module is used to estimate information about the three-dimensional spatial extent.

[0217] In some embodiments, a first key point of the first three-dimensional pose and a second key point of the second three-dimensional pose are obtained; based on the first key point and the second key point, the viewpoint of the second three-dimensional pose and the viewpoint of the first three-dimensional pose are corrected to be consistent; based on the difference between the corrected first three-dimensional pose and the second three-dimensional pose, at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained.

[0218] In some embodiments, the viewpoint of the second three-dimensional pose is corrected to the viewpoint of the first three-dimensional pose to generate a third three-dimensional pose, so as to train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the third three-dimensional pose and the first three-dimensional pose.

[0219] In some embodiments, a first up-dimensional network module is used to estimate a first keypoint and a second keypoint.

[0220] In some embodiments, the first keypoint is located at the center of the first three-dimensional pose, and the second keypoint is located at the center of the second three-dimensional pose.

[0221] In some embodiments, the two-dimensional pose of the target is estimated using a first two-dimensional detection module based on any two-dimensional image; and the first three-dimensional pose is estimated using a second upscaling network module based on the two-dimensional pose of the target.

[0222] In some embodiments, a two-dimensional feature point heatmap of any first two-dimensional image is estimated based on any first two-dimensional image; and a two-dimensional pose is estimated based on the two-dimensional feature point heatmap.

[0223] In some embodiments, the two-dimensional pose is estimated by performing soft-argmax processing on the two-dimensional feature point heatmap.

[0224] In some embodiments, the second up-dimensional network module includes a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer.

[0225] In some embodiments, the MSE loss and the two-dimensional reprojection loss are determined based on the difference between the first three-dimensional pose and the second three-dimensional pose; at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained based on the weighted average of the MSE loss and the two-dimensional reprojection loss until the training termination condition is met.

[0226] In some embodiments, the target includes a human target, and the multiple feature points of the human target include multiple key points.

[0227] In some embodiments, the training device 6a further includes a display device 62a for displaying at least one of a first three-dimensional pose or a second three-dimensional pose.

[0228] Figure 6b shows a block diagram of a three-dimensional pose estimation apparatus according to some embodiments of the present disclosure.

[0229] As shown in Figure 6b, in some embodiments, the three-dimensional pose estimation device 6b includes: a training device 6a for a machine learning model in any of the above embodiments, configured to train a first three-dimensional pose estimation module for a machine learning model; and at least one processor 61b, which is configured to: acquire a two-dimensional image containing the target; and estimate the three-dimensional pose of the target based on the two-dimensional image using the trained first three-dimensional pose estimation module.

[0230] In some embodiments, processor 61b is configured to: obtain the usage requirements of the machine learning model; and convert the machine learning model into a format that matches the usage requirements.

[0231] In some embodiments, the processor 61b is configured to: in response to a user's setting operation, set the deployment environment and model parameters for the machine learning model.

[0232] In some embodiments, the display device 62a is used to display three-dimensional poses.

[0233] Figure 7 shows a block diagram of some embodiments of the electronic devices disclosed herein.

[0234] As shown in FIG7, the electronic device 7 of this embodiment includes: a memory 71 and a processor 72 coupled to the memory 71. The processor 72 is configured to execute a training method for a machine learning model or a three-dimensional pose estimation method in any embodiment of the present disclosure based on instructions stored in the memory 71.

[0235] The memory 71 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory stores, for example, the operating system, application programs, boot loader, database, and other programs.

[0236] Figure 8 shows a block diagram of some other embodiments of the electronic device disclosed herein.

[0237] As shown in FIG8, the electronic device 8 of this embodiment includes: a memory 810 and a processor 820 coupled to the memory 810. The processor 820 is configured to execute the training method of the machine learning model or the estimation method of the three-dimensional pose in any of the foregoing embodiments based on the instructions stored in the memory 810.

[0238] The memory 810 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, the operating system, application programs, boot loader, and other programs.

[0239] Electronic device 8 may also include input / output interface 830, network interface 840, storage interface 850, etc. These interfaces 830, 840, 850, as well as memory 810 and processor 820, can be connected via, for example, a bus 860. Specifically, input / output interface 830 provides a connection interface for input / output devices such as monitors, mice, keyboards, touchscreens, microphones, and speakers. Network interface 840 provides a connection interface for various networked devices. Storage interface 850 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0240] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0241] The present disclosure has now been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0242] The methods and systems of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the specific order described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0243] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A training apparatus for a machine learning model, comprising at least one processor, said at least one processor being configured to: Acquire multiple two-dimensional images and the machine learning model to be trained, wherein the multiple two-dimensional images contain a target, and the machine learning model includes a first three-dimensional pose estimation module and a second three-dimensional pose estimation module; Based on any two-dimensional image, the first three-dimensional pose of the target is estimated using the first three-dimensional pose estimation module; The second three-dimensional pose of the target is estimated using the second three-dimensional pose estimation module based on at least two two-dimensional images, wherein the viewpoints of the at least two two-dimensional images are different from each other; Based on the difference between the first 3D pose and the second 3D pose, train at least one of the first 3D pose estimation module or the second 3D pose estimation module.

2. The training device according to claim 1, wherein, The target includes multiple feature points, and estimating the second three-dimensional pose of the target using the second three-dimensional pose estimation module based on at least two two-dimensional images includes: Based on the at least two two-dimensional images, estimate the information of the three-dimensional spatial range of each of the multiple feature points of the target; The second three-dimensional pose is estimated based on the information of the three-dimensional spatial range.

3. The training device according to claim 2, wherein, The step of estimating the three-dimensional spatial range of each of the multiple feature segments of the target based on the at least two two-dimensional images includes: Estimate the two-dimensional feature point heatmap of each of the at least two two-dimensional images; Based on the two-dimensional feature point heatmap, information about the three-dimensional spatial range is estimated.

4. The training device according to claim 2, wherein, The step of estimating the second three-dimensional pose based on the information of the three-dimensional spatial range includes: Based on the information of the three-dimensional spatial range, estimate the three-dimensional feature point heatmap; The second three-dimensional pose is estimated based on the three-dimensional feature point heatmap.

5. The training device according to claim 4, wherein, The step of estimating the second three-dimensional pose based on the three-dimensional feature point heatmap includes: The second three-dimensional pose is estimated by performing soft-argmax processing on the three-dimensional feature point heatmap.

6. The training device according to claim 2, wherein, The three-dimensional spatial range includes the cubic spatial range.

7. The training device according to claim 2, wherein, The step of estimating the three-dimensional spatial range of each of the multiple feature points of the target based on the at least two two-dimensional images includes: The information of the three-dimensional spatial range is estimated using a three-dimensional image structure module.

8. The training apparatus according to any one of claims 1-7, wherein, The step of training at least one of the first 3D pose estimation module or the second 3D pose estimation module based on the difference between the first 3D pose and the second 3D pose includes: Obtain the first key point of the first three-dimensional pose and the second key point of the second three-dimensional pose; Based on the first key point and the second key point, the viewpoint of the second three-dimensional pose and the viewpoint of the first three-dimensional pose are corrected to be consistent. Based on the difference between the corrected first 3D pose and the second 3D pose, train at least one of the first 3D pose estimation module or the second 3D pose estimation module.

9. The training apparatus according to claim 8, wherein, The step of correcting the viewpoint of the second three-dimensional pose to be consistent with the viewpoint of the first three-dimensional pose based on the first key point and the second key point includes: The viewpoint of the second three-dimensional pose is corrected to that of the first three-dimensional pose to generate a third three-dimensional pose, so that at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module can be trained based on the difference between the third three-dimensional pose and the first three-dimensional pose.

10. The training apparatus according to claim 8, wherein, The process of obtaining the first key point of the first three-dimensional pose and the second key point of the second three-dimensional pose includes: The first key point and the second key point are estimated using the first up-dimensional network module.

11. The training apparatus according to claim 8, wherein, The first key point is located at the center of the first three-dimensional pose, and the second key point is located at the center of the second three-dimensional pose.

12. The training apparatus according to any one of claims 1-7, wherein, The step of estimating the first three-dimensional pose of the target using the first three-dimensional pose estimation module based on any two-dimensional image includes: Based on any one of the two-dimensional images, the two-dimensional pose of the target is estimated using the first two-dimensional detection module; Based on the two-dimensional pose of the target, the first three-dimensional pose is estimated using a second up-dimensional network module.

13. The training apparatus according to claim 12, wherein, The step of estimating the two-dimensional pose of the target using the first two-dimensional detection module based on any of the first two-dimensional images includes: Based on any one of the first two-dimensional images, estimate a two-dimensional feature point heatmap of the first two-dimensional image; The two-dimensional pose is estimated based on the two-dimensional feature point heatmap.

14. The training apparatus according to claim 13, wherein, The step of estimating the two-dimensional pose based on the two-dimensional feature point heatmap includes: The two-dimensional pose is estimated by performing soft-argmax processing on the two-dimensional feature point heatmap.

15. The training apparatus according to claim 12, wherein, The second upscaling network module includes a batch normalization layer, a random deactivation dropout layer, a rectified linear layer, and a residual connection layer.

16. The training apparatus according to any one of claims 1-7, wherein, The step of training at least one of the first 3D pose estimation module or the second 3D pose estimation module based on the difference between the first 3D pose and the second 3D pose includes: Based on the difference between the first three-dimensional pose and the second three-dimensional pose, the mean square error loss and the two-dimensional reprojection loss are determined; Based on the weighted average of the MSE loss and the two-dimensional reprojection loss, train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module until the training termination condition is met.

17. The training apparatus according to any one of claims 1-7, wherein, The target includes a human target, and the multiple feature points of the human target include multiple key points.

18. The training apparatus according to any one of claims 1-7, further comprising: A display device for displaying at least one of a first three-dimensional pose or a second three-dimensional pose.

19. The estimating apparatus according to any one of claims 1-7, wherein, The at least two two-dimensional images are acquired based on the perspective and projection matrix of multiple synchronized shooting devices.

20. A three-dimensional pose estimation device, comprising: The training apparatus for the machine learning model according to any one of claims 1-19 is configured as a first three-dimensional pose estimation module for training the machine learning model; as well as At least one processor, said at least one processor being configured to: Obtain a two-dimensional image containing the target; Based on the two-dimensional image, the three-dimensional pose of the target is estimated using the first three-dimensional pose estimation module that has completed training.

21. The estimation apparatus according to claim 20, wherein, The processor is configured to: Obtain the usage requirements of the machine learning model; The machine learning model is converted into a format that matches the usage requirements.

22. The estimation apparatus according to claim 20, wherein, The processor is configured to: In response to the user's settings operation, the deployment environment and model parameters of the machine learning model are configured.

23. The estimating apparatus according to any one of claims 20-22, wherein, The display device is used to display the three-dimensional pose.

24. A method for training a machine learning model, comprising: Based on any two-dimensional image containing the target, the first three-dimensional pose of the target is estimated using the first three-dimensional pose estimation module of the machine learning model; The second three-dimensional pose of the target is estimated using the second three-dimensional pose estimation module of the machine learning model based on at least two two-dimensional images containing the target, wherein the viewpoints of the plurality of second two-dimensional images are different from each other; Based on the difference between the first 3D pose and the second 3D pose, train at least one of the first 3D pose estimation module or the second 3D pose estimation module.

25. A method for estimating three-dimensional pose, comprising: According to the training method of the machine learning model according to claim 24, the first three-dimensional pose estimation module of the machine learning model is trained; Based on a two-dimensional image containing the target, the three-dimensional pose of the target is estimated using the first three-dimensional pose estimation module.

26. An electronic device comprising: Memory; and A processor coupled to the memory, the processor being configured to execute the training method of the machine learning model of claim 24 or the estimation method of the three-dimensional pose of claim 25 based on instructions stored in the memory device.

27. A non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method of the machine learning model as described in claim 24 or the estimation method of the three-dimensional pose as described in claim 25.

Images