Machine learning model training apparatus and method, and three-dimensional pose estimation apparatus

Abstract

The present disclosure relates to the technical field of computers, and relates to a machine learning model training apparatus and method, and a three-dimensional pose estimation apparatus. The training apparatus comprises at least one processor, the at least one processor being configured to: acquire a plurality of two-dimensional images and a machine learning model to be trained, the plurality of two-dimensional images comprising a target, and the machine learning model comprising a first three-dimensional pose estimation module and a second three-dimensional pose estimation module; according to any one two-dimensional image, use the first three-dimensional pose estimation module to estimate a first three-dimensional pose of the target; according to at least two two-dimensional images, use the second three-dimensional pose estimation module to estimate a second three-dimensional pose of the target, the viewing angles of the at least two two-dimensional images being different from each other; and training at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module according to the difference between the first three-dimensional pose and the second three-dimensional pose.

Description

Machine learning model training device, method and three-dimensional posture estimation device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the application with CN application number 202310691946.6 and application date June 12, 2023, and claims its priority. The disclosed content of the CN application is hereby introduced as a whole into this application. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a training device for a machine learning model, a training method for a machine learning model, a three-dimensional posture estimation device, a three-dimensional posture estimation method, an electronic device, and a non-volatile computer-readable storage medium. Background Art

[0004] Methods for estimating 3D human pose from a monocular perspective have attracted widespread attention in the field of computer vision and have spawned numerous related applications. For example, estimating 3D human pose through machine learning has led to a variety of applications, including human-computer interaction, action recognition, and virtual reality.

[0005] In related technologies, with the great success of deep learning models and increasingly complex datasets, deep CNNs (Convolutional Neural Networks) can be applied to estimate 3D human pose in a monocular camera setting.

[0006] Summary of the Invention

[0007] According to some embodiments of the present disclosure, a training device for a machine learning model is provided, comprising at least one processor, wherein the at least one processor is configured to: acquire multiple two-dimensional images and a machine learning model to be trained, wherein the multiple two-dimensional images contain a target, and the machine learning model comprises a first three-dimensional posture estimation module and a second three-dimensional posture estimation module; estimate the first three-dimensional posture of the target using the first three-dimensional posture estimation module based on any two-dimensional image; estimate the second three-dimensional posture of the target using the second three-dimensional posture estimation module based on at least two two-dimensional images, wherein the viewing angles of at least two two-dimensional images are different; and train at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module based on the difference between the first three-dimensional posture and the second three-dimensional posture.

[0008] In some embodiments, the target includes multiple feature points, and estimating the second three-dimensional pose of the target using a second three-dimensional pose estimation module based on at least two two-dimensional images includes: estimating information about the three-dimensional spatial range of each of the multiple feature points of the target based on at least two two-dimensional images; and estimating the second three-dimensional pose based on the information about the three-dimensional spatial range.

[0009] In some embodiments, estimating the three-dimensional spatial range information of each of the multiple feature nodes of the target based on at least two two-dimensional images includes: estimating the two-dimensional feature point heat map of each of the at least two two-dimensional images; and estimating the three-dimensional spatial range information based on the two-dimensional feature point heat map.

[0010] In some embodiments, estimating the second three-dimensional pose based on the information of the three-dimensional space range includes: estimating a three-dimensional feature point heat map based on the information of the three-dimensional space range; and estimating the second three-dimensional pose based on the three-dimensional feature point heat map.

[0011] In some embodiments, estimating the second three-dimensional pose based on the three-dimensional feature point heat map includes: estimating the second three-dimensional pose by performing soft-argmax processing on the three-dimensional feature point heat map.

[0012] In some embodiments, the three-dimensional spatial extent comprises a cubic spatial extent.

[0013] In some embodiments, estimating the 3D spatial range of each of the plurality of feature points of the target based on at least two 2D images includes estimating the 3D spatial range using a 3D Pictorial Structures (3DPS) module.

[0014] In some embodiments, based on the difference between the first three-dimensional posture and the second three-dimensional posture, training at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module includes: obtaining a first key point of the first three-dimensional posture and a second key point of the second three-dimensional posture; based on the first key point and the second key point, correcting the perspective of the second three-dimensional posture and the perspective of the first three-dimensional posture to a consistent perspective; based on the difference between the corrected first three-dimensional posture and the second three-dimensional posture, training at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module.

[0015] In some embodiments, based on the first key point and the second key point, correcting the perspective of the second three-dimensional pose and the perspective of the first three-dimensional pose to a consistent perspective includes: correcting the perspective of the second three-dimensional pose to the perspective of the first three-dimensional pose to generate a third three-dimensional pose, so as to train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the third three-dimensional pose and the first three-dimensional pose.

[0016] In some embodiments, obtaining a first key point of the first three-dimensional pose and a second key point of the second three-dimensional pose includes: estimating the first key point and the second key point by using a first dimensionality-raising network module.

[0017] In some embodiments, the first keypoint is located at a center position of the first three-dimensional pose, and the second keypoint is located at a center position of the second three-dimensional pose.

[0018] In some embodiments, estimating the first three-dimensional posture of the target using a first three-dimensional posture estimation module based on any two-dimensional image includes: estimating the two-dimensional posture of the target using a first two-dimensional detection module based on any two-dimensional image; and estimating the first three-dimensional posture based on the two-dimensional posture of the target using a second dimensionality-increasing network module.

[0019] In some embodiments, estimating the two-dimensional posture of the target using the first two-dimensional detection module based on any first two-dimensional image includes: estimating the two-dimensional feature point heat map of any first two-dimensional image based on any first two-dimensional image; estimating the two-dimensional posture based on the two-dimensional feature point heat map.

[0020] In some embodiments, estimating the two-dimensional pose based on the two-dimensional feature point heat map includes: estimating the two-dimensional pose by performing a soft-argmax process on the two-dimensional feature point heat map.

[0021] In some embodiments, the second dimension-raising network module includes a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer.

[0022] In some embodiments, based on the difference between the first three-dimensional pose and the second three-dimensional pose, training at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module includes: determining the MSE (Mean Squared Error) loss and the two-dimensional reprojection loss based on the difference between the first three-dimensional pose and the second three-dimensional pose; and training at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the weighted average of the MSE loss and the two-dimensional reprojection loss until the training end condition is met.

[0023] In some embodiments, the target includes a human target, and the multiple feature points of the human target include multiple joint points.

[0024] In some embodiments, the training device further comprises: a display device for displaying at least one of the first three-dimensional gesture or the second three-dimensional gesture.

[0025] According to other embodiments of the present disclosure, a three-dimensional posture estimation device is provided, comprising: a training device for the machine learning model in any of the above embodiments, configured to train a first three-dimensional posture estimation module of the machine learning model; and at least one processor, wherein the at least one processor is configured to: obtain a two-dimensional image containing a target; and estimate the three-dimensional posture of the target based on the two-dimensional image using the trained first three-dimensional posture estimation module.

[0026] In some embodiments, the processor is configured to: obtain usage requirements of the machine learning model; and convert the machine learning model into a format that matches the usage requirements.

[0027] In some embodiments, the processor is configured to: set the deployment environment and model parameters of the machine learning model in response to the user's setting operation.

[0028] In some embodiments, the display device is used to display the three-dimensional gesture.

[0029] According to some further embodiments of the present disclosure, a method for training a machine learning model is provided, comprising: estimating a first three-dimensional posture of the target using a first three-dimensional posture estimation module of the machine learning model based on any two-dimensional image containing the target; estimating a second three-dimensional posture of the target using a second three-dimensional posture estimation module of the machine learning model based on at least two two-dimensional images containing the target, the perspectives of the multiple second two-dimensional images being different from each other; and training at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module based on the difference between the first three-dimensional posture and the second three-dimensional posture.

[0030] According to some further embodiments of the present disclosure, a three-dimensional posture estimation method is provided, including: training a first three-dimensional posture estimation module of a machine learning model according to the training method of the machine learning model in any of the above embodiments; and estimating the three-dimensional posture of the target using the first three-dimensional posture estimation module based on a two-dimensional image containing the target.

[0031] According to some further embodiments of the present disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the training method of the machine learning model or the three-dimensional posture estimation method in any of the above-mentioned embodiments based on instructions stored in the memory device.

[0032] According to some further embodiments of the present disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the training method of the machine learning model or the three-dimensional posture estimation method in any of the above embodiments is implemented.

[0033] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of this application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0035] FIG1 shows a flowchart of a method for training a machine learning model according to some embodiments of the present disclosure;

[0036] FIG2 is a schematic diagram showing a method for training a machine learning model according to other embodiments of the present disclosure;

[0037] FIG3 is a schematic diagram illustrating a method for training a machine learning model according to some embodiments of the present disclosure;

[0038] FIG4 is a schematic diagram showing a method for estimating a three-dimensional pose according to some embodiments of the present disclosure;

[0039] Figures 5a to 5e illustrate schematic diagrams of a machine learning model development and deployment platform according to some embodiments of the present disclosure;

[0040] FIG6 a shows a block diagram of a training apparatus for a machine learning model according to some embodiments of the present disclosure;

[0041] FIG6 b shows a block diagram of a device for estimating a three-dimensional pose according to some embodiments of the present disclosure;

[0042] FIG7 shows a block diagram of some embodiments of an electronic device of the present disclosure;

[0043] FIG8 shows a block diagram of other embodiments of the electronic device of the present disclosure. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0045] Unless otherwise specifically stated, the relative arrangement of the parts and steps, the numerical expressions and the numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to actual proportional relationships. The techniques, methods and equipment known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods and equipment should be considered as part of the authorization specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as being merely exemplary and not as limitations. Therefore, other examples of the exemplary embodiments may have different values. It should be noted that similar numbers and letters represent similar items in the following figures, and therefore, once an item is defined in one figure, it does not need to be further discussed in subsequent figures.

[0046] The previously described methods for estimating 3D pose from a monocular camera view using neural networks face several challenges.

[0047] First, most typical neural network models require a large amount of annotated 3D human pose data for training. These methods rely on labeled 3D skeletons as 3D pose annotations for depth prediction, collected using marker-based motion capture systems. However, marker-based motion capture systems are expensive. In other words, using marker-based motion capture systems to annotate 3D poses as ground truth is an expensive process.

[0048] Secondly, the mathematical theory of projecting 2D joints into 3D space is well-documented, and neural networks are an approximation of this projection. Therefore, over-reliance on real 3D data may lead to overfitting problems.

[0049] Thirdly, methods trained on a monocular camera view can lead to depth ambiguity. Since multiple different 3D skeletons can be projected into the same 2D pose in a specific camera view, estimating 3D pose from a monocular view can result in strange 3D poses.

[0050] To address the above technical issues, this paper proposes a self-supervised model. This model requires only unlabeled multi-view data, without the need for annotated training data. For example, this paper can train a network based on geometric knowledge, without the need for labeled 3D poses.

[0051] For example, the present disclosure constructs a pipeline that follows two steps: a 2D skeleton estimator from multiple camera views, and a 2D-to-3D pose upscaling network that outputs an estimated 3D pose. The skeleton estimator can estimate the 2D coordinates or 2D heat map of each feature point that constitutes the skeleton model of the object.

[0052] For example, the present disclosure also utilizes multi-view consistency constraints to better converge the captured 2D pose to the 3D pose. For incomplete or erroneous 2D detection situations, the multi-view setting also solves the technical problem of depth ambiguity.

[0053] For example, this disclosure adopts the confidence of 2D joints to integrate losses from different viewpoints to mitigate the impact of noise caused by self-occlusion problems.

[0054] To address the above technical issues, this paper proposes a self-supervised training method in a multi-view training setting. By using a self-supervised method instead of explicit 3D ground truth annotations, training can be completed using only 2D pose estimates generated by a 2D pose estimator as input.

[0055] This not only overcomes the depth ambiguity problem through a multi-view setting, but also solves the problem of incomplete or erroneous 2D detection by leveraging information from other viewpoints.

[0056] For example, since information from different perspectives and monocular settings have a wider range of application scenarios, multi-view settings are more advantageous when training models. The present disclosure can only apply the multi-view setting during training and the monocular setting during testing, thereby combining the advantages of both paradigms.

[0057] For example, the architecture of this disclosure blends the outputs of two branches of a neural network. The first branch takes a single image (from a single camera view) as input and generates a 3D pose in 3D space; the second branch takes multiple images from different camera views and outputs an estimated 3D pose. The method provided by this disclosure allows all estimated 3D poses from both branches to be projected to any camera view.

[0058] In some embodiments, the method provided by the present disclosure may include two phases. The first phase uses a 2D pose estimator to estimate the 2D human pose. The second phase elevates these 2D estimates to 3D space. Combining the camera's geometric information, the 3D pose from the second branch, and the rotation matrix of the viewpoint from the first branch, the desired 3D pose is obtained in the first camera coordinate system through rotation and other processing.

[0059] That is, all 3D poses in the real-world coordinate system should be exactly the same, so each acquired pose can be projected back into each camera's own 3D camera coordinate system. In addition, the 3D pose is reprojected to each 2D camera view through the camera matrix, and a reprojection loss can be defined for each 3D to 2D projection.

[0060] In some embodiments, the present disclosure proposes a two-branch self-supervised approach in a multi-view training setting to train a 2D to 3D upscaling network without requiring 3D annotation of the data. The disclosed technical solution relies solely on geometric knowledge to construct a supervisory signal, resulting in better generalization capabilities.

[0061] In some embodiments, the present disclosure proposes a recurrent view training scheme that is effective in leveraging multi-view consistency information and constraining the estimated 3D pose during training. This overcomes depth ambiguity and can leverage information from other views to address incomplete or erroneous 2D detections. Furthermore, the present disclosure utilizes the joint 2D confidence of different camera views to mitigate self-occlusion issues.

[0062] In some embodiments, the present disclosure proposes a volumetric 3DPS network based on a voxel-based approach and 3DPS. The network is capable of exploring a sufficient state space where a volume cube is generated by grid sampling; the cube is then sent to a voxel-based neural network to obtain a 3D pose.

[0063] The inventors of this disclosure have discovered the following problem with the aforementioned related technologies: machine learning models require a large number of labeled data samples, resulting in high pose estimation costs. In light of this, this disclosure proposes a technical solution for training machine learning models. For example, the technical solution of this disclosure can be implemented through the following embodiments.

[0064] FIG1 shows a flowchart of a method for training a machine learning model according to some embodiments of the present disclosure.

[0065] As shown in FIG1 , in step 110, a plurality of two-dimensional images and a machine learning model to be trained are obtained. The plurality of two-dimensional images contain a target. The machine learning model includes a first three-dimensional pose estimation module and a second three-dimensional pose estimation module. For example, the target includes a human target, and the plurality of feature points of the human target include a plurality of joint points.

[0066] In some embodiments, the machine learning model includes a first 3D pose estimation module and a second 3D pose estimation module. The first 3D pose estimation module includes a first 2D detection module and a second dimensionality-increasing network module; the second 3D pose estimation module includes a second 2D detection module and a 3DPS module. The first 2D detection module and the second 2D detection module can be the same or different.

[0067] In some embodiments, when there are four camera views, there are four synchronized camera views and camera projection matrices, and all cameras capture two-dimensional images of the same object and the same scene.

[0068] For example, the multiple two-dimensional images processed by the machine learning model are obtained according to the perspectives and projection matrices of multiple synchronized shooting devices to ensure that the multiple two-dimensional images remain synchronized in time, thereby improving the training effect and processing accuracy of the machine learning model.

[0069] In step 120, a first three-dimensional pose estimation module is used to estimate a first three-dimensional pose of the target based on any two-dimensional image. For example, the two-dimensional image is acquired from a first shooting angle by a monocular camera.

[0070] This allows the first 3D pose estimation module to be trained without labeling the training samples. The trained first 3D pose estimation module can accurately estimate the 3D pose of an object in a monocular camera scenario without requiring multiple cameras, thus reducing costs.

[0071] In some embodiments, the first three-dimensional pose may include at least one of a three-dimensional heat map or three-dimensional coordinates.

[0072] For example, when using a trained first 3D pose estimation module for estimation, the first 3D pose can be output to the user in the form of 3D coordinates, thereby providing more accurate information. The user can then perform subsequent operations such as behavior recognition and action warning based on the 3D coordinates.

[0073] For example, during the training of the first 3D pose estimation module, the first 3D pose can be output in the form of a 3D heat map. The heat map can represent the range of feature points rather than a fixed point, providing the system with a higher error tolerance, thereby improving the training effect of the first 3D pose estimation module.

[0074] In some embodiments, the first two-dimensional detection module is used to estimate the two-dimensional pose of the target based on any two-dimensional image.

[0075] For example, a 2D pose estimator is used to predict a 2D pose. For each frame of a 2D image, the 2D image can be cropped based on the bounding box of the target in the 2D image detected by the 2D detector.

[0076] For example, the cropped 2D image from the perspective of the first camera among the 4 cameras is represented by I, and the cropped 2D images from the perspectives of the other 3 cameras are represented by Representation: Each cropped 2D image is input into the 2D pose estimator to estimate the 2D pose X of the target.

[0077] For example, the backbone of a 2D pose estimator is represented by f and the weight parameter is θ. The 2D pose estimator can be composed of two parts: a global network for roughly predicting the 2D pose and a fine network for refining the joints.

[0078] In some embodiments, a 2D feature point heat map of any first 2D image is estimated based on any first 2D image, and a 2D pose is estimated based on the 2D feature point heat map, for example, by performing a soft-argmax process on the 2D feature point heat map to estimate the 2D pose.

[0079] For example, the heatmap H of I can be estimated by a 2D pose estimator: H = f(I; θ)

[0080] For example, the two-dimensional pose X of the target can be estimated by H. The argmax algorithm can be used to estimate the two-dimensional pose X. The argmax algorithm can obtain the joint with the highest probability by calculating the maximum point of each heat map.

[0081] However, the argmax algorithm may cut off the gradient flow, making it impossible to train the 2D pose estimator. To address this technical issue, for example, soft-argmax can be used to apply to the heat map H instead of argmax:

[0082] H k is the heat map of the kth joint of the target captured by the first camera, and Ω is the domain of the heat map. The soft-argmax uses the probability G corresponding to position i k (i) Weight position i to obtain the weighted coordinates of position i; add up all the weighted coordinates to obtain the two-dimensional coordinates of each joint as the two-dimensional posture of the target.

[0083] In the above embodiment, the soft-argmax process can ensure that the gradient flow is transferred from the output three-dimensional pose to the input two-dimensional image, so that the machine learning model does not cut off the gradient flow, thereby improving the accuracy of pose estimation.

[0084] In some embodiments, the first three-dimensional pose is estimated using a second dimension-raising network module based on the two-dimensional pose of the target. For example, the N detected 2D joints are represented as Use 3D dimensionality-raising network to predict 3D pose

[0085] In some embodiments, the second dimension-raising network module includes a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer.

[0086] For example, the dimension-raising network w vThe goal is to estimate the joint position of the target in 3D space given only 2D input. The dimension-raising network may include a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer through a residual connection. The input layer of the dimension-raising network may input the coordinates of N (a positive integer such as 17) human joints and apply them to a fully connected layer with 1024 output channels; then, it is processed using a residual connection layer consisting of 4 modules connected through residual connections, each module may be composed of two fully connected layers; then, it is processed using a batch normalization layer, a rectified linear unit, and a dropout layer; finally, the final feature output by a residual module is input into a linear layer to obtain the three-dimensional posture Y: Y = w v (X)

[0087] In the above embodiment, a 2D pose estimation network is used to predict 2D poses and heatmaps for multiple input frames. The confidence scores of all 2D feature points are then upgraded to 3D feature point confidence scores to integrate losses from different viewpoints. This mitigates the impact of noise caused by self-occlusion, thereby improving pose estimation accuracy.

[0088] In step 130 , a second 3D pose estimation module is used to estimate a second 3D pose of the target based on at least two 2D images, where the viewing angles of the at least two 2D images are different from each other.

[0089] For example, the second three-dimensional posture may include at least one of a three-dimensional heat map of the target or three-dimensional coordinates of each feature point of the target.

[0090] In some embodiments, the at least two two-dimensional images can be acquired from different shooting angles by a monocular camera, or by multiple cameras. For example, the two two-dimensional images can be acquired from a second angle and a third angle, respectively, by a monocular camera, or by one binocular camera from a second angle and the other from a third angle, respectively. The second angle and the third angle are different.

[0091] In some embodiments, the object includes multiple feature points. Based on at least two two-dimensional images, information about the three-dimensional spatial range within which each of the multiple feature points of the object resides is estimated; and based on the information about the three-dimensional spatial range, a second three-dimensional pose is estimated. For example, the three-dimensional spatial range includes a cubic spatial range, a spherical spatial range, etc.

[0092] In some embodiments, the three-dimensional spatial range corresponding to the feature point may be a spatial range centered on the feature point. For example, a 3DPS network module may be used to estimate a three-dimensional heat map of the feature point's location, and the three-dimensional spatial range corresponding to the feature point may be determined based on the three-dimensional heat map.

[0093] For example, the size of the three-dimensional spatial range can be determined by a preset threshold. A heat threshold can be used to determine the range of the three-dimensional heat map where the heat of the feature point is greater than the heat threshold as the three-dimensional spatial range of the feature point.

[0094] In some embodiments, a 2D feature point heat map is estimated for each of at least two 2D images, and information about a 3D spatial range is estimated based on the 2D feature point heat map, for example, using a 3DPS module.

[0095] For example, receiving input 2D images from 3 different camera perspectives And output a 2D heat map for each 2D image of the view

[0096] In some embodiments, the 3DPS module is a multi-view 3D pose prediction method that can explore sufficient state space and generate 3D human body part candidates through grid sampling; then, predict the 2D pose through a 2D detector; and generate the 3D pose using 2D priors and maximum likelihood estimation.

[0097] In some embodiments, to improve the accuracy and robustness of 3DPS, a volumetric algorithm can be combined with 3DPS to obtain a volumetric 3DPS network to estimate the 3D pose.

[0098] For example, using a volumetric 3DPS network that receives a 2D heatmap as input Output 3D pose after processing

[0099] For example, the 3DPS module can be used to sample the 3D cubes around the joints of the human body instead of sampling the 3D joints. The 3DPS method is applied to the 2D heat map corresponding to the joint k under the view angle c. After processing, the volume cube corresponding to joint k under view angle c is obtained as the cube space range:

[0100] is the volume cube generated based on the k-th joint heatmap of the c-th camera view, and T represents the processing of the 3DPS method. For example, the size of the volume cube can be 32×32×32.

[0101] The volume cubes at each perspective can be added together to get the sum of the target volume cubes

[0102] represents the sum of the volume cubes of joint k under multiple viewpoints c.

[0103] In some embodiments, a three-dimensional feature point heat map is estimated based on information of the three-dimensional spatial range; and a second three-dimensional posture is estimated based on the three-dimensional feature point heat map.

[0104] For example, the sum of the volume cubes is fed into a learnable volume convolutional neural network p q The architecture of the volume convolutional neural network can be similar to the voxel-to-voxel network architecture. A three-dimensional feature point heat map H can be obtained. 3D : H 3D =p q (V sum )

[0105] V sum is the sum of the volume cubes of each joint k The splicing result of , q represents the weight of the voxel-to-voxel network.

[0106] In some embodiments, the second 3D pose is estimated by performing a soft-argmax process on the 3D feature point heat map.

[0107] For example, applying soft-argmax to the heatmap H 3D , get the three-dimensional pose

[0108] Represents the heat map of the kth joint of the estimated target in three-dimensional space, Ω represents the domain; thus, the position of the kth three-dimensional joint can be obtained As a 3D pose:

[0109] In the above embodiment, a volumetric algorithm (i.e., estimating the 3D spatial range corresponding to a feature point) and 3DPS are combined to sample the 3D spatial range around the feature point of the target and send this 3D spatial range to a voxel-based neural network to obtain the 3D pose. This improves the accuracy and robustness of 3DPS, thereby improving the accuracy of pose estimation.

[0110] In step 140 , at least one of the first 3D pose estimation module or the second 3D pose estimation module is trained based on the difference between the first 3D pose and the second 3D pose.

[0111] In the above embodiment, the machine learning model of the present disclosure is trained in a multi-view setting using a self-supervised approach, so that the machine learning model benefits from information from different viewpoints, thereby improving the accuracy of pose estimation.

[0112] However, because the training data lacks true 3D pose annotations, the 3D poses generated by the two branches are unlikely to be at the same absolute 3D position, potentially preventing training from converging. To address this technical issue, a pre-training solution can be applied to help localize the 3D poses, ensuring that the 3D poses generated by the two branches are at the same absolute 3D position. For example, this technical solution can be implemented through the following embodiments.

[0113] In some embodiments, a first key point of a first three-dimensional pose and a second key point of a second three-dimensional pose are obtained; based on the first key point and the second key point, the perspective of the second three-dimensional pose and the perspective of the first three-dimensional pose are corrected to a consistent perspective; and based on the difference between the corrected first three-dimensional pose and the second three-dimensional pose, at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained.

[0114] In some embodiments, the first dimensionality-raising network module is used to estimate a first key point and a second key point. For example, the first key point is located at the center of the first three-dimensional pose, and the second key point is located at the center of the second three-dimensional pose. For example, the human pelvis can be used as the key point at the center. For example, the first key point and the second key point can be estimated based on the feature points.

[0115] For example, a first dimension-raising network module different from a second dimension-raising network module can be used to predict the center position of a three-dimensional posture. The center first dimension-raising network module and the second dimension-raising network module can have the same network structure but different weight parameters. The first dimension-raising network module can include u γ and Two parts, u γ Used to predict the first key point Y under the first camera perspective pelvis , Used to predict the second keypoints from the second, third and fourth camera views randomly selected Y pelvis =u γ (X)

[0116] is the predicted 2D pose at a randomly chosen viewpoint. The predicted keypoints can then be used to localize the 3D pose, and during training, the 3D pose is moved to the predicted position to ensure that the viewpoint of the second 3D pose is aligned with the viewpoint of the first 3D pose.

[0117] In some embodiments, the perspective of the second three-dimensional pose is corrected to the perspective of the first three-dimensional pose to generate a third three-dimensional pose, so as to train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the third three-dimensional pose and the first three-dimensional pose.

[0118] For example, the correspondence between feature points on the second 3D pose and the first 3D pose can be determined. Single-line alignment can then be performed based on the positions of the first and second key points, as well as the edges of the second 3D pose and the edges of the first 3D pose, which have corresponding feature points. This ensures that the perspective of the second 3D pose is consistent with that of the first 3D pose, thereby improving the accuracy of pose estimation.

[0119] In some embodiments, the MSE loss and the two-dimensional reprojection loss are determined based on the difference between the first three-dimensional pose and the second three-dimensional pose; and at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained based on the weighted average of the MSE loss and the two-dimensional reprojection loss until the training end condition is met.

[0120] For example, two different loss functions, 3D MSE loss and 2D reprojection loss, can be applied to train a machine learning model. The reprojection loss can further constrain the consistency of multiple views, thereby improving the performance of the machine learning model.

[0121] This is because, in the absence of depth information, a 2D detector corresponds to multiple possibilities of 3D poses in different directions. 2D reprojection can help constrain it to the correct direction by leveraging information from other viewpoints, thereby improving the performance of machine learning models.

[0122] For example, the MSE loss can be:

[0123] Y k is the first 3D posture of feature point k, is the second 3D posture of feature point k.

[0124] For example, the 2D reprojection loss is: P c =[p1 p2 p3] T .

[0125] p1 p2 p3 represent the coordinate transformation information required for coordinate transformation.

[0126] In some embodiments, the total loss for training the machine learning model is:

[0127] α is a weight coefficient set as needed to adjust the importance of the two losses in training.

[0128] FIG2 shows a schematic diagram of a method for training a machine learning model according to some other embodiments of the present disclosure.

[0129] As shown in FIG2 , in step 210 , the first three-dimensional posture estimation module of the machine learning model is trained according to the training method in any one of the above embodiments.

[0130] In step 220, a two-dimensional image containing the target is acquired; and based on the two-dimensional image, the three-dimensional pose of the target is estimated using the trained first three-dimensional pose estimation module.

[0131] In some embodiments, usage requirements of a machine learning model are obtained; and the machine learning model is converted into a format that matches the usage requirements.

[0132] In some embodiments, in response to a user's setting operation, the deployment environment and model parameters of the machine learning model are set.

[0133] In some embodiments, a three-dimensional gesture is displayed.

[0134] In some embodiments, the target includes multiple feature points, and information about the three-dimensional spatial range of each of the multiple feature points of the target is estimated based on at least two two-dimensional images; and a second three-dimensional posture is estimated based on the information about the three-dimensional spatial range.

[0135] In some embodiments, a two-dimensional feature point heat map is estimated for each of at least two two-dimensional images; and information about a three-dimensional spatial range is estimated based on the two-dimensional feature point heat map.

[0136] In some embodiments, a three-dimensional feature point heat map is estimated based on information of the three-dimensional spatial range; and a second three-dimensional posture is estimated based on the three-dimensional feature point heat map.

[0137] In some embodiments, the second 3D pose is estimated by performing a soft-argmax process on the 3D feature point heat map.

[0138] In some embodiments, the three-dimensional spatial extent comprises a cubic spatial extent.

[0139] In some embodiments, the 3DPS module is used to estimate the three-dimensional spatial range information.

[0140] In some embodiments, a first key point of a first three-dimensional pose and a second key point of a second three-dimensional pose are obtained; based on the first key point and the second key point, the perspective of the second three-dimensional pose and the perspective of the first three-dimensional pose are corrected to a consistent perspective; and based on the difference between the corrected first three-dimensional pose and the second three-dimensional pose, at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained.

[0141] In some embodiments, the perspective of the second three-dimensional pose is corrected to the perspective of the first three-dimensional pose to generate a third three-dimensional pose, so as to train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the third three-dimensional pose and the first three-dimensional pose.

[0142] In some embodiments, the first key point and the second key point are estimated using a first dimension-raising network module.

[0143] In some embodiments, the first keypoint is located at a center position of the first three-dimensional pose, and the second keypoint is located at a center position of the second three-dimensional pose.

[0144] In some embodiments, based on any two-dimensional image, a first two-dimensional detection module is used to estimate the two-dimensional posture of the target; based on the two-dimensional posture of the target, a second dimension-raising network module is used to estimate the first three-dimensional posture.

[0145] In some embodiments, a two-dimensional feature point heat map of any first two-dimensional image is estimated based on any first two-dimensional image; and a two-dimensional posture is estimated based on the two-dimensional feature point heat map.

[0146] In some embodiments, the 2D pose is estimated by performing a soft-argmax process on the 2D feature point heatmap.

[0147] In some embodiments, the second dimension-raising network module includes a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer.

[0148] In some embodiments, the MSE loss and the two-dimensional reprojection loss are determined based on the difference between the first three-dimensional pose and the second three-dimensional pose; and at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained based on the weighted average of the MSE loss and the two-dimensional reprojection loss until the training end condition is met.

[0149] In some embodiments, the target includes a human target, and the multiple feature points of the human target include multiple joint points.

[0150] In some embodiments, at least one of the first three-dimensional gesture or the second three-dimensional gesture is displayed.

[0151] FIG3 shows a schematic diagram of a method for training a machine learning model according to some embodiments of the present disclosure.

[0152] As shown in Figure 3, the machine learning model consists of two branches. The first branch inputs a single 2D image from the first camera perspective (from one camera view) and generates a first 3D pose in 3D space. The second branch inputs three 2D images from the other three camera perspectives and outputs an estimated second 3D pose. The second 3D pose can be rotated to the first camera perspective to achieve multi-view consistency loss. This ensures that the 3D poses estimated from different perspectives have the same skeleton before geometric transformation. The predicted 3D pose can be reprojected to each camera perspective to obtain additional reprojection error.

[0153] In some embodiments, the machine learning model includes the two-dimensional detector, the dimension-raising network, and the volumetric 3DPS network in Figure 3. The two-dimensional detector includes the first two-dimensional detection module and the second two-dimensional detection module in any embodiment; the dimension-raising network includes the second dimension-raising network module in any embodiment; and the volumetric 3DPS network includes the 3DPS module in any embodiment.

[0154] For example, the first three-dimensional pose estimation module includes the two-dimensional detector and the dimension-raising network in Figure 3; the second three-dimensional pose estimation module includes the two-dimensional detector and the volume 3DPS network in Figure 3.

[0155] In some embodiments, a 2D detector is applied to predict a 2D pose and a 2D heat map of 4 input 2D image frames; then, a dimensionality-raising network is used to raise the confidence of each 2D feature point to 3D.

[0156] Figure 3 shows the network architecture for four camera views. There are four synchronized camera views and camera projection matrices, and all cameras capture the same object and scene. First, a 2D detector is used to obtain the 2D pose. For each 2D image frame, the bounding box of the object detected by the 2D detector is used to crop the 2D image.

[0157] For example, the cropped 2D image from the perspective of the first camera among the 4 cameras is represented by I, and the cropped 2D images from the perspectives of the other 3 cameras are represented by Each cropped 2D image is fed into a 2D pose estimator to estimate the 2D pose X of the target. For example, the backbone of the 2D pose estimator is represented by f, with weight parameters θ. The 2D pose estimator can be composed of two parts: a global network for roughly estimating the 2D pose and a fine network for refining the joints.

[0158] As shown in Figure 3, the training process has two branches. In the first branch, Represented as N detected 2D joints; then, a 3D dimensionality-raising network is used to predict the 3D pose In the other branch, it receives the input 2D images from 3 camera views and outputs a 2D heatmap for each view Then, the volumetric 3DPS network is used to receive the heatmap input and output the 3D pose. The heat map can be estimated by a two-dimensional detector: H = f(I; θ),

[0159] The 2D pose X for the first branch is estimated by H. The argmax algorithm can be used to estimate the 2D pose X. The argmax algorithm can obtain the joint with the highest probability by calculating the maximum point of each heat map.

[0160] However, the argmax algorithm may cut off the gradient flow, making it impossible to train the 2D pose estimator. To address this technical issue, for example, soft-argmax can be used to apply to the heat map H instead of argmax:

[0161] In some embodiments, the three-dimensional pose can be estimated with zero as the center, and Y and The value of is the 3D position relative to the fixed root joint. The predicted 3D pose is in its own pose coordinate system during training, and can then be The 3D loss is obtained by rotating to the first camera coordinate; the reprojection loss of the pose of each view is obtained by reprojecting the global 3D pose into the second external space.

[0162] For example, the Lifting Network v The goal is to estimate the joint position of the target in 3D space given only 2D input. The dimension-raising network may include a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer through a residual connection. The input layer of the dimension-raising network may input the coordinates of N (a positive integer such as 17) human joints and apply them to a fully connected layer with 1024 output channels; then, it is processed using a residual connection layer consisting of 4 modules connected through residual connections, each module may be composed of two fully connected layers; then, it is processed using a batch normalization layer, a rectified linear unit, and a dropout layer; finally, the final feature output by a residual module is input into a linear layer to obtain the three-dimensional posture Y: Y = w v (X)

[0163] The 3DPS module is a multi-view 3D pose prediction method that can explore sufficient state space and generate 3D human part candidates through grid sampling; then, it predicts 2D pose through a 2D detector; and generates 3D pose using 2D priors and maximum likelihood estimation.

[0164] In order to improve the accuracy and robustness of 3DPS, the volumetric algorithm and 3DPS can be combined to obtain a volumetric 3DPS network to estimate the 3D pose. The 3DPS module can be used to sample the 3D cubes around the joints of the human body instead of sampling the 3D joints. The 3DPS method is applied to the 2D heat map corresponding to the joint k under the view angle c. After processing, the volume cube corresponding to joint k under view angle c is obtained as the cube space range:

[0165] is the volume cube generated based on the k-th joint heatmap of the c-th camera view, and T represents the processing of the 3DPS method. For example, the size of the volume cube can be 32×32×32.

[0166] The volume cubes at each perspective can be added together to get the sum of the target volume cubes

[0167] represents the sum of the volume cubes of joint k under multiple viewpoints c.

[0168] The sum of the volume cubes is input into the learnable volume convolutional neural network p q The architecture of the volume convolutional neural network can be similar to the voxel-to-voxel network architecture. A three-dimensional feature point heat map H can be obtained. 3D : H 3D =p q (V sum )

[0169] V sum is the sum of the volume cubes of each joint k The splicing result of , q represents the weight of the voxel-to-voxel network.

[0170] Apply soft-argmax to the heatmap H 3D , get the three-dimensional pose

[0171] Represents the heat map of the kth joint of the estimated target in three-dimensional space, Ω represents the domain; thus, the position of the kth three-dimensional joint can be obtained As a 3D pose:

[0172] For example, using the L2 norm between the 3D poses predicted from different viewpoints It ensures consistency across multiple views. If the prediction result is accurate, the loss should be zero. This is because the same object captured from different camera views should have the same absolute 3D pose in 3D space.

[0173] However, this solution does not work well for more than two views. During training, the L2-norm loss for the view of 2D image 1 versus the view of 2D image 2 conflicts with the L2-norm loss for the view of 2D image 2 versus the view of 2D image 3. This is because the dimensionality-raising network learns a 3D pose that is only applicable to one viewpoint, without the depth information from the camera viewpoints.

[0174] To address these technical issues, 3D poses generated from different camera perspectives can be converted into the same 3D pose. For example, a cyclic perspective training scheme can be used. The machine learning model in Figure 3 consists of two branches: the first branch is based on a single camera perspective, followed by a dimensionality-increasing network; the second branch is based on three camera perspectives, followed by a volumetric 3DPS network. By randomly swapping the input 2D images of these four perspectives, all possible 3D perspective combinations can be trained to address the multi-perspective consistency issue.

[0175] For example, after one round of training using the training method in any of the above embodiments, one two-dimensional image can be selected from at least two two-dimensional images in the previous round of training as the first two-dimensional image, the remaining one of the at least two two-dimensional images can be determined as the second two-dimensional image, and any two-dimensional image in the previous round of training can be determined as the second two-dimensional image; based on the first two-dimensional image, the first three-dimensional posture estimation module is used to estimate the first three-dimensional posture of the target in this round of training; based on the second two-dimensional image, the second three-dimensional posture estimation module is used to estimate the second three-dimensional posture of the target in this round of training; based on the difference between the first three-dimensional posture in this round of training and the second three-dimensional posture in this round of training, at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module is trained.

[0176] In the above embodiment, the machine learning model of the present disclosure is trained in a multi-view setting using a self-supervised approach, so that the machine learning model benefits from information from different viewpoints, thereby improving the accuracy of pose estimation.

[0177] However, because the training data lacks true 3D pose annotations, the 3D poses generated by the two branches are unlikely to be at the same absolute 3D position, potentially preventing training from converging. To address this technical issue, a pre-training solution can be applied to help localize the 3D poses, ensuring that the 3D poses generated by the two branches are at the same absolute 3D position.

[0178] For example, a first dimension-raising network module different from a second dimension-raising network module can be used to predict the center position of a three-dimensional posture. The center first dimension-raising network module and the second dimension-raising network module can have the same network structure but different weight parameters. The first dimension-raising network module can include u γ and Two parts, u γ Used to predict the first key point Y under the first camera perspective pelvis , Used to predict the second keypoints from the second, third and fourth camera views randomly selected Y pelvis =u γ (X)

[0179] The predicted keypoints can then be used to localize the 3D pose, and during training the 3D pose can be moved to the predicted position to ensure that the perspective of the second 3D pose is corrected to the same perspective as the first 3D pose.

[0180] For example, the double arrow between the first 3D pose and the third 3D pose in FIG3 may represent training a machine learning model by comparing the differences between the first 3D pose and the third 3D pose.

[0181] For example, soft-argmax can be used to process the 3D heatmap generated by the dimensionality-raising network to obtain the first 3D pose. Soft-argmax processing ensures that the gradient flow from the output 3D pose is passed to the input 2D image, preventing the machine learning model from cutting off the gradient flow.

[0182] Two different loss functions, 3D MSE loss and 2D reprojection loss, can be applied to train machine learning models. Reprojection loss can further constrain the consistency of multiple views, thereby improving the performance of machine learning models.

[0183] This is because, in the absence of depth information, a 2D detector corresponds to multiple possibilities of 3D poses in different orientations, and 2D reprojection can help constrain it to the correct orientation by leveraging information from other viewpoints, thereby improving the performance of machine learning models.

[0184] For example, the MSE loss can be:

[0185] Y k is the first 3D posture of feature point k, is the second 3D posture of feature point k.

[0186] For example, the 2D reprojection loss is: P c =[p1 p2 p3] T .

[0187] p1 p2 p3 represent the coordinate transformation information required for coordinate transformation.

[0188] In some embodiments, the total loss for training the machine learning model is:

[0189] α is a weight coefficient set as needed to adjust the importance of the two losses in training.

[0190] For example, multi-view 3D human pose estimation methods can use input 2D images from multiple different viewpoints, thus achieving better performance than using a single monocular camera view.

[0191] In some embodiments, during the training phase, a cyclic view can be used to assist in the refinement of the single view process and the voxel V2V 3DPS (i.e., volumetric 3DPS) neural network. The volumetric 3DPS network includes a 3DPS module and a V2V network for estimating the cube in which the feature points are located.

[0192] For example, a multi-view setup can be used only during the training phase, while a single-view setup is required during the testing phase. This allows the training of a 3D pose estimation model without the need for 3D data annotation.

[0193] For example, the reprojection loss helps constrain multi-view information during training. The voxel-based V2V 3DPS neural network and the recurrent perspective-based training process avoid technical issues caused by data labeling, enabling label-free training and reducing training costs and barriers to entry.

[0194] FIG4 shows a schematic diagram of a method for estimating a three-dimensional pose according to some embodiments of the present disclosure.

[0195] As shown in Figure 4, the machine learning model feeds each 2D input image into a bounding box detector to obtain cropped 2D images, which are then fed into the backbone network of the 2D pose estimator. The estimated 2D pose is then fed into the dimension-increasing network to obtain the estimated 3D pose. For example, the multi-view setting is only used during the training phase; the monocular camera setting is followed during the testing phase.

[0196] Figures 5a to 5e show schematic diagrams of a machine learning model development and deployment platform according to some embodiments of the present disclosure.

[0197] As shown in Figure 5a, upload multi-view pedestrian data images (i.e., 2D images) on the data upload page and, according to the corresponding naming rules, establish the time sequence relationship between each pedestrian data image. During data upload, in addition to the time sequence information transmitted by the camera itself, no manual data annotation is required to support subsequent training and test deployment.

[0198] For example, the data upload page includes a task details area and a data file list area. The task details area can include relevant information about the training task, and the data file list area can include relevant information about the uploaded data.

[0199] As shown in Figure 5b, on the model building page, you can upload and write the code for the machine learning model to be trained using a notebook. The uploaded code can include the code for setting up the training and testing models and related processes.

[0200] For example, the model upload and writing area of ​​the model building page includes information about the notebook and the development environment.

[0201] As shown in Figure 5c, on the model training page, the corresponding parameters of the model are designed and adjusted accordingly. When training the model, a multi-branch perspective is needed to assist the training process, so that the training itself can achieve better results, and no training annotation is required.

[0202] For example, the model configuration area of ​​the model training page includes configuration functions for training tasks and lithium acid resources.

[0203] As shown in Figure 5d, on the model conversion page, users can convert trained models to facilitate subsequent deployment. For example, a machine learning model can be converted from PyTorch to an architecture like ONNX. During model deployment, model conversion and accuracy correction can be used to meet diverse customer needs.

[0204] For example, the Conversion Configuration area of ​​the Model Conversion page includes information about the machine learning model and conversion configuration capabilities.

[0205] As shown in Figure 5e, on the model deployment page, select the trained model and set the relevant deployment environment and related parameters to ensure that the model can run correctly when called.

[0206] For example, a machine learning model can use the HTTP (Hypertext Transfer Protocol) mode. The test input to the model can be a single image of a pedestrian from a single viewpoint. The trained model can then identify the 3D pose of the pedestrian in the image, providing a basis for subsequent action recognition or behavioral analysis.

[0207] During the deployment phase, this model only needs to use the single-process weight model in the test file to adapt to the needs of the real environment and reduce the hardware costs and conditions required for operation.

[0208] For example, the Model Deployment area of ​​the Model Deployment page includes information about the model and about the service to be processed.

[0209] Figure 6a shows a block diagram of a training device for a machine learning model according to some embodiments of the present disclosure.

[0210] As shown in Figure 6a, the training device 6a of the machine learning model includes at least one processor 61a, and the at least one processor 61a is configured to: obtain multiple two-dimensional images and a machine learning model to be trained, the multiple two-dimensional images contain targets, and the machine learning model includes a first three-dimensional posture estimation module and a second three-dimensional posture estimation module; based on any two-dimensional image, use the first three-dimensional posture estimation module to estimate the first three-dimensional posture of the target; based on at least two two-dimensional images, use the second three-dimensional posture estimation module to estimate the second three-dimensional posture of the target, and the perspectives of at least two two-dimensional images are different from each other; based on the difference between the first three-dimensional posture and the second three-dimensional posture, train at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module.

[0211] In some embodiments, the target includes multiple feature points, and information about the three-dimensional spatial range of each of the multiple feature points of the target is estimated based on at least two two-dimensional images; and a second three-dimensional posture is estimated based on the information about the three-dimensional spatial range.

[0212] In some embodiments, a two-dimensional feature point heat map is estimated for each of at least two two-dimensional images; and information about a three-dimensional spatial range is estimated based on the two-dimensional feature point heat map.

[0213] In some embodiments, a three-dimensional feature point heat map is estimated based on information of the three-dimensional spatial range; and a second three-dimensional posture is estimated based on the three-dimensional feature point heat map.

[0214] In some embodiments, the second 3D pose is estimated by performing a soft-argmax process on the 3D feature point heat map.

[0215] In some embodiments, the three-dimensional spatial extent comprises a cubic spatial extent.

[0216] In some embodiments, the 3DPS module is used to estimate the three-dimensional spatial range information.

[0217] In some embodiments, a first key point of a first three-dimensional pose and a second key point of a second three-dimensional pose are obtained; based on the first key point and the second key point, the perspective of the second three-dimensional pose and the perspective of the first three-dimensional pose are corrected to a consistent perspective; and based on the difference between the corrected first three-dimensional pose and the second three-dimensional pose, at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained.

[0218] In some embodiments, the perspective of the second three-dimensional pose is corrected to the perspective of the first three-dimensional pose to generate a third three-dimensional pose, so as to train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the third three-dimensional pose and the first three-dimensional pose.

[0219] In some embodiments, the first key point and the second key point are estimated using a first dimension-raising network module.

[0220] In some embodiments, the first keypoint is located at a center position of the first three-dimensional pose, and the second keypoint is located at a center position of the second three-dimensional pose.

[0221] In some embodiments, based on any two-dimensional image, a first two-dimensional detection module is used to estimate the two-dimensional posture of the target; based on the two-dimensional posture of the target, a second dimension-raising network module is used to estimate the first three-dimensional posture.

[0222] In some embodiments, a two-dimensional feature point heat map of any first two-dimensional image is estimated based on any first two-dimensional image; and a two-dimensional posture is estimated based on the two-dimensional feature point heat map.

[0223] In some embodiments, the 2D pose is estimated by performing a soft-argmax process on the 2D feature point heatmap.

[0224] In some embodiments, the second dimension-raising network module includes a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer.

[0225] In some embodiments, the MSE loss and the two-dimensional reprojection loss are determined based on the difference between the first three-dimensional pose and the second three-dimensional pose; and at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained based on the weighted average of the MSE loss and the two-dimensional reprojection loss until the training end condition is met.

[0226] In some embodiments, the target includes a human target, and the multiple feature points of the human target include multiple joint points.

[0227] In some embodiments, the training device 6a further includes a display device 62a for displaying at least one of the first three-dimensional gesture or the second three-dimensional gesture.

[0228] FIG6 b shows a block diagram of a device for estimating a three-dimensional pose according to some embodiments of the present disclosure.

[0229] As shown in Figure 6b, in some embodiments, the three-dimensional posture estimation device 6b includes: the training device 6a of the machine learning model in any of the above embodiments, configured to train the first three-dimensional posture estimation module of the machine learning model; and at least one processor 61b, at least one processor 61b is configured to: obtain a two-dimensional image containing the target; and estimate the three-dimensional posture of the target based on the two-dimensional image using the trained first three-dimensional posture estimation module.

[0230] In some embodiments, the processor 61b is configured to: obtain usage requirements of the machine learning model; and convert the machine learning model into a format that matches the usage requirements.

[0231] In some embodiments, the processor 61b is configured to: set the deployment environment and model parameters of the machine learning model in response to the user's setting operation.

[0232] In some embodiments, the display device 62a is used to display three-dimensional gestures.

[0233] FIG7 shows a block diagram of some embodiments of the electronic device of the present disclosure.

[0234] As shown in Figure 7, the electronic device 7 of this embodiment includes: a memory 71 and a processor 72 coupled to the memory 71, and the processor 72 is configured to execute the training method of the machine learning model or the three-dimensional posture estimation method in any embodiment of the present disclosure based on the instructions stored in the memory 71.

[0235] The memory 71 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0236] FIG8 shows a block diagram of other embodiments of the electronic device of the present disclosure.

[0237] As shown in Figure 8, the electronic device 8 of this embodiment includes: a memory 810 and a processor 820 coupled to the memory 810, and the processor 820 is configured to execute the training method of the machine learning model or the three-dimensional posture estimation method in any of the aforementioned embodiments based on the instructions stored in the memory 810.

[0238] The memory 810 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs.

[0239] The electronic device 8 may also include an input / output interface 830, a network interface 840, a storage interface 850, and the like. These interfaces 830, 840, 850, as well as the memory 810 and the processor 820, may be connected, for example, via a bus 860. The input / output interface 830 provides a connection interface for input / output devices such as a display, mouse, keyboard, touch screen, microphone, and speakers. The network interface 840 provides a connection interface for various networked devices. The storage interface 850 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0240] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0241] So far, the present invention has been described in detail. In order to avoid obscuring the concept of the present invention, some details well known in the art have not been described. Based on the above description, those skilled in the art can fully understand how to implement the technical solutions disclosed herein.

[0242] The methods and systems of the present disclosure may be implemented in many ways. For example, the methods and systems of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0243] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art will appreciate that the above examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art will appreciate that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A training device for a machine learning model, comprising at least one processor, wherein the at least one processor is configured to: Acquire a plurality of two-dimensional images and the machine learning model to be trained, wherein the plurality of two-dimensional images contain targets, and the machine learning model includes a first three-dimensional posture estimation module and a second three-dimensional posture estimation module; According to any one of the two-dimensional images, using the first three-dimensional posture estimation module, estimating a first three-dimensional posture of the target; estimating a second three-dimensional posture of the target using the second three-dimensional posture estimation module according to at least two two-dimensional images, wherein the viewing angles of the at least two two-dimensional images are different from each other; At least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained according to a difference between the first three-dimensional pose and the second three-dimensional pose.

2. The training device according to claim 1, wherein: The target includes a plurality of feature points, and estimating the second three-dimensional posture of the target by using the second three-dimensional posture estimation module according to at least two two-dimensional images includes: According to the at least two two-dimensional images, estimating information of a three-dimensional spatial range where each of the plurality of feature points of the target is located; The second three-dimensional posture is estimated according to the information of the three-dimensional space range.

3. The training device according to claim 2, wherein: The information of estimating the three-dimensional space range of each of the multiple characteristic nodes of the target according to the at least two two-dimensional images includes: estimating a two-dimensional feature point heat map for each of the at least two two-dimensional images; According to the two-dimensional feature point heat map, information of the three-dimensional space range is estimated.

4. The training device according to claim 2, wherein: The estimating the second three-dimensional posture according to the information of the three-dimensional space range includes: According to the information of the three-dimensional spatial range, a three-dimensional feature point heat map is estimated; The second three-dimensional posture is estimated according to the three-dimensional feature point heat map.

5. The training device according to claim 4, wherein: The estimating the second three-dimensional posture according to the three-dimensional feature point heat map includes: The second three-dimensional posture is estimated by performing soft-argmax processing on the three-dimensional feature point heat map.

6. The training device according to claim 2, wherein: The three-dimensional space range includes a cubic space range.

7. The training device according to claim 2, wherein: The information of estimating the three-dimensional space range of each of the multiple feature points of the target according to the at least two two-dimensional images includes: The three-dimensional image structure module is used to estimate the information of the three-dimensional space range.

8. The training device according to any one of claims 1 to 7, wherein: The training of at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module according to the difference between the first three-dimensional posture and the second three-dimensional posture comprises: Acquire a first key point of the first three-dimensional posture and a second key point of the second three-dimensional posture; Correcting the perspective of the second three-dimensional posture and the perspective of the first three-dimensional posture to be consistent with each other according to the first key point and the second key point; At least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained according to the difference between the corrected first three-dimensional pose and the second three-dimensional pose.

9. The training device according to claim 8, wherein: The step of correcting the perspective of the second three-dimensional posture and the perspective of the first three-dimensional posture to be consistent with each other according to the first key point and the second key point comprises: The perspective of the second three-dimensional pose is corrected to the perspective of the first three-dimensional pose to generate a third three-dimensional pose, so as to train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module according to the difference between the third three-dimensional pose and the first three-dimensional pose.

10. The training device according to claim 8, wherein: The acquiring of the first key point of the first three-dimensional posture and the second key point of the second three-dimensional posture comprises: The first key point and the second key point are estimated by using a first dimension-raising network module.

11. The training device according to claim 8, wherein: The first key point is located at a center position of the first three-dimensional posture, and the second key point is located at a center position of the second three-dimensional posture.

12. The training device according to any one of claims 1 to 7, wherein: The estimating the first three-dimensional posture of the target by using the first three-dimensional posture estimation module according to any two-dimensional image comprises: According to any one of the two-dimensional images, using a first two-dimensional detection module to estimate the two-dimensional posture of the target; The first three-dimensional posture is estimated by using a second dimension-raising network module according to the two-dimensional posture of the target.

13. The training device according to claim 12, wherein: The estimating the two-dimensional posture of the target by using a first two-dimensional detection module according to any one of the first two-dimensional images comprises: According to any one of the first two-dimensional images, estimating a two-dimensional feature point heat map of any one of the first two-dimensional images; The two-dimensional posture is estimated according to the two-dimensional feature point heat map.

14. The training device according to claim 13, wherein: The estimating the two-dimensional posture according to the two-dimensional feature point heat map comprises: The two-dimensional posture is estimated by performing soft-argmax processing on the two-dimensional feature point heat map.

15. The training device according to claim 12, wherein: The second dimension-raising network module includes a batch normalization layer, a random inactivation dropout layer, a rectified linear layer and a residual connection layer.

16. The training device according to any one of claims 1 to 7, wherein: The training of at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module according to the difference between the first three-dimensional posture and the second three-dimensional posture comprises: determining a mean square error loss and a two-dimensional reprojection loss based on a difference between the first three-dimensional pose and the second three-dimensional pose; According to the weighted mean of the MSE loss and the two-dimensional reprojection loss, at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained until a training end condition is met.

17. The training device according to any one of claims 1 to 7, wherein: The target includes a human target, and the multiple feature points of the human target include multiple joint points.

18. The training device according to any one of claims 1 to 7, further comprising: The display device is used to display at least one of the first three-dimensional gesture or the second three-dimensional gesture.

19. The estimation device according to any one of claims 1 to 7, wherein: The at least two two-dimensional images are acquired according to the viewing angles and projection matrices of a plurality of synchronized shooting devices.

20. A three-dimensional posture estimation device, comprising: A training device for a machine learning model according to any one of claims 1 to 19, configured to train a first three-dimensional pose estimation module of the machine learning model; as well as at least one processor, the at least one processor being configured to: Acquire a two-dimensional image containing the target; The three-dimensional posture of the target is estimated according to the two-dimensional image using a trained first three-dimensional posture estimation module.

21. The estimation device according to claim 20, wherein: The processor is configured to: Obtaining usage requirements for the machine learning model; Convert the machine learning model into a format that matches the usage requirements.

22. The estimation device according to claim 20, wherein: The processor is configured to: In response to the user's setting operation, the deployment environment and model parameters of the machine learning model are set.

23. The estimation device according to any one of claims 20 to 22, wherein: The display device is used to display the three-dimensional posture.

24. A method for training a machine learning model, comprising: According to any two-dimensional image containing the target, estimating a first three-dimensional posture of the target using a first three-dimensional posture estimation module of the machine learning model; estimating a second three-dimensional posture of the target using a second three-dimensional posture estimation module of the machine learning model according to at least two two-dimensional images containing the target, wherein the viewing angles of the plurality of second two-dimensional images are different from each other; At least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained according to a difference between the first three-dimensional pose and the second three-dimensional pose.

25. A method for estimating a three-dimensional posture, comprising: According to the training method of the machine learning model according to claim 24, training a first three-dimensional posture estimation module of the machine learning model; The three-dimensional posture of the target is estimated using the first three-dimensional posture estimation module according to the two-dimensional image containing the target.

26. An electronic device comprising: Memory; and A processor coupled to the memory, the processor being configured to execute the training method of the machine learning model described in claim 24 or the three-dimensional posture estimation method described in claim 25 based on instructions stored in the memory device.

27. A non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method of the machine learning model as described in claim 24 or the three-dimensional posture estimation method as described in claim 25.

Images