Device and method for training machine learning model, and device for estimating three-dimensional attitude

Abstract

The invention relates to a training device and method of a machine learning model and a three-dimensional attitude estimation device, and relates to the technical field of computers. The training device comprises at least one processor, the at least one processor is configured to obtain a plurality of two-dimensional images and a to-be-trained machine learning model, the plurality of two-dimensional images comprise a target, and the machine learning model comprises a first three-dimensional attitude estimation module and a second three-dimensional attitude estimation module; estimating a first three-dimensional attitude of the target by using a first three-dimensional attitude estimation module according to any two-dimensional image; according to the at least two two-dimensional images, estimating a second three-dimensional attitude of the target by using a second three-dimensional attitude estimation module, the viewing angles of the at least two two-dimensional images being different from each other; and training at least one of the first three-dimensional attitude estimation module or the second three-dimensional attitude estimation module according to the difference between the first three-dimensional attitude and the second three-dimensional attitude.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a training device for a machine learning model, a training method for a machine learning model, a three-dimensional posture estimation device, a three-dimensional posture estimation method, an electronic device, and a non-volatile computer-readable storage medium. Background Art

[0002] The 3D human pose estimation method under monocular view has received extensive attention in the field of computer vision and has derived many related applications. For example, the 3D human pose estimation through machine learning can be derived into various applications, including human-computer interaction, action recognition and virtual reality.

[0003] In the related art, with the great success of deep learning models and increasingly complex datasets, deep CNNs (Convolutional Neural Networks) can be applied to estimate 3D human pose in a monocular camera setting. Summary of the invention

[0004] According to some embodiments of the present disclosure, there is provided a training device for a machine learning model, comprising at least one processor, wherein the at least one processor is configured to: obtain multiple two-dimensional images and a machine learning model to be trained, wherein the multiple two-dimensional images contain a target, and the machine learning model comprises a first three-dimensional posture estimation module and a second three-dimensional posture estimation module; estimate the first three-dimensional posture of the target using the first three-dimensional posture estimation module based on any two-dimensional image; estimate the second three-dimensional posture of the target using the second three-dimensional posture estimation module based on at least two two-dimensional images, wherein the viewing angles of at least two two-dimensional images are different; and train at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module based on the difference between the first three-dimensional posture and the second three-dimensional posture.

[0005] In some embodiments, the target includes multiple feature points, and estimating the second three-dimensional posture of the target using a second three-dimensional posture estimation module based on at least two two-dimensional images includes: estimating information about the three-dimensional space range of each of the multiple feature points of the target based on at least two two-dimensional images; estimating the second three-dimensional posture based on the information about the three-dimensional space range.

[0006] In some embodiments, estimating the three-dimensional spatial range information of each of multiple feature nodes of the target based on at least two two-dimensional images includes: estimating the two-dimensional feature point heat map of each of at least two two-dimensional images; estimating the three-dimensional spatial range information based on the two-dimensional feature point heat map.

[0007] In some embodiments, estimating the second three-dimensional posture according to the information of the three-dimensional space range includes: estimating a three-dimensional feature point heat map according to the information of the three-dimensional space range; estimating the second three-dimensional posture according to the three-dimensional feature point heat map.

[0008] In some embodiments, estimating the second three-dimensional pose according to the three-dimensional feature point heat map includes: estimating the second three-dimensional pose by performing soft-argmax processing on the three-dimensional feature point heat map.

[0009] In some embodiments, the three-dimensional spatial extent includes a cubic spatial extent.

[0010] In some embodiments, estimating the three-dimensional space range information of each of the multiple feature points of the target based on at least two two-dimensional images includes: estimating the three-dimensional space range information using a 3DPS (3D Pictorial Structures) module.

[0011] In some embodiments, based on the difference between the first three-dimensional posture and the second three-dimensional posture, training at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module includes: acquiring a first key point of the first three-dimensional posture and a second key point of the second three-dimensional posture; based on the first key point and the second key point, correcting the perspective of the second three-dimensional posture and the perspective of the first three-dimensional posture to a consistent perspective; based on the difference between the corrected first three-dimensional posture and the second three-dimensional posture, training at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module.

[0012] In some embodiments, based on the first key point and the second key point, correcting the perspective of the second three-dimensional posture and the perspective of the first three-dimensional posture to a consistent perspective includes: correcting the perspective of the second three-dimensional posture to the perspective of the first three-dimensional posture to generate a third three-dimensional posture, so as to train at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module based on the difference between the third three-dimensional posture and the first three-dimensional posture.

[0013] In some embodiments, obtaining a first key point of the first three-dimensional posture and a second key point of the second three-dimensional posture includes: estimating the first key point and the second key point by using a first dimensionality-raising network module.

[0014] In some embodiments, the first key point is located at a center position of the first three-dimensional pose, and the second key point is located at a center position of the second three-dimensional pose.

[0015] In some embodiments, estimating the first three-dimensional posture of the target using a first three-dimensional posture estimation module according to any two-dimensional image includes: estimating the two-dimensional posture of the target using a first two-dimensional detection module according to any two-dimensional image; estimating the first three-dimensional posture according to the two-dimensional posture of the target using a second dimensionality-increasing network module.

[0016] In some embodiments, estimating the two-dimensional posture of the target using the first two-dimensional detection module based on any first two-dimensional image includes: estimating the two-dimensional feature point heat map of any first two-dimensional image based on any first two-dimensional image; estimating the two-dimensional posture based on the two-dimensional feature point heat map.

[0017] In some embodiments, estimating the two-dimensional pose according to the two-dimensional feature point heat map includes: estimating the two-dimensional pose by performing soft-argmax processing on the two-dimensional feature point heat map.

[0018] In some embodiments, the second dimension-raising network module includes a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer.

[0019] In some embodiments, based on the difference between the first three-dimensional posture and the second three-dimensional posture, training at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module includes: determining the MSE (Mean Squared Error) loss and the two-dimensional reprojection loss based on the difference between the first three-dimensional posture and the second three-dimensional posture; training at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module based on the weighted mean of the MSE loss and the two-dimensional reprojection loss until a training end condition is met.

[0020] In some embodiments, the target includes a human target, and the plurality of feature points of the human target includes a plurality of joint points.

[0021] In some embodiments, the training device further comprises: a display device for displaying at least one of the first three-dimensional gesture or the second three-dimensional gesture.

[0022] According to other embodiments of the present disclosure, a three-dimensional posture estimation device is provided, comprising: a training device for the machine learning model in any of the above embodiments, configured to train a first three-dimensional posture estimation module of the machine learning model; and at least one processor, wherein the at least one processor is configured to: acquire a two-dimensional image containing a target; and estimate the three-dimensional posture of the target based on the two-dimensional image using the trained first three-dimensional posture estimation module.

[0023] In some embodiments, the processor is configured to: obtain usage requirements of the machine learning model; and convert the machine learning model into a format that matches the usage requirements.

[0024] In some embodiments, the processor is configured to: set the deployment environment and model parameters of the machine learning model in response to the user's setting operation.

[0025] In some embodiments, the display device is used to display the three-dimensional gesture.

[0026] According to some further embodiments of the present disclosure, a method for training a machine learning model is provided, comprising: estimating a first three-dimensional posture of the target using a first three-dimensional posture estimation module of the machine learning model based on any two-dimensional image containing the target; estimating a second three-dimensional posture of the target using a second three-dimensional posture estimation module of the machine learning model based on at least two two-dimensional images containing the target, the viewing angles of the multiple second two-dimensional images being different from each other; and training at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module based on the difference between the first three-dimensional posture and the second three-dimensional posture.

[0027] According to some further embodiments of the present disclosure, a three-dimensional posture estimation method is provided, comprising: training a first three-dimensional posture estimation module of a machine learning model according to the training method of a machine learning model in any of the above embodiments; estimating the three-dimensional posture of the target using the first three-dimensional posture estimation module based on a two-dimensional image containing the target.

[0028] According to some further embodiments of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the training method of the machine learning model or the three-dimensional posture estimation method in any one of the above-mentioned embodiments based on instructions stored in the memory device.

[0029] According to some further embodiments of the present disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the training method of the machine learning model or the three-dimensional posture estimation method in any of the above-mentioned embodiments is implemented.

[0030] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0032] Figure 1 A flowchart showing a method for training a machine learning model according to some embodiments of the present disclosure;

[0033] Figure 2A schematic diagram showing a training method for a machine learning model according to some other embodiments of the present disclosure;

[0034] Figure 3 A schematic diagram showing a method for training a machine learning model according to some embodiments of the present disclosure;

[0035] Figure 4 A schematic diagram showing a method for estimating a three-dimensional posture according to some embodiments of the present disclosure;

[0036] Figures 5a to 5e A schematic diagram showing a machine learning model development and deployment platform according to some embodiments of the present disclosure;

[0037] Figure 6a A block diagram showing a training apparatus for a machine learning model according to some embodiments of the present disclosure;

[0038] Figure 6b A block diagram showing a three-dimensional posture estimation device according to some embodiments of the present disclosure;

[0039] Figure 7 A block diagram showing some embodiments of the electronic device of the present disclosure;

[0040] Figure 8 A block diagram showing some other embodiments of the electronic device of the present disclosure. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0042] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, numerical expressions and numerical values ​​do not limit the scope of the present disclosure. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The technology, methods and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, methods and equipment should be regarded as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0043] The previously described approach of estimating 3D pose from a monocular camera view using neural networks faces several challenges.

[0044] First, most typical neural network models require a large amount of annotated 3D human pose data for training. These methods rely on labeled 3D skeletons as 3D pose annotations for depth prediction, which are collected by marker-based motion capture systems. However, marker-based motion capture systems are expensive. In other words, annotating 3D poses as ground truth using marker-based motion capture systems is a costly process.

[0045] Secondly, the mathematical theory of projecting 2D joints into 3D space is well-documented, and neural networks are themselves an approximation of this projection. Therefore, over-reliance on 3D real data may lead to overfitting problems.

[0046] Thirdly, methods trained on a monocular camera view can lead to depth ambiguity issues. Since multiple different 3D skeletons can be projected into the same 2D pose in a particular camera view, estimating 3D pose from a monocular view can result in strange 3D poses.

[0047] In response to the above technical problems, the present disclosure proposes a self-supervised model. The model only requires unlabeled multi-view data and does not require annotated training data. For example, the present disclosure can train a network based on geometric knowledge without the need for labeled 3D poses.

[0048] For example, the present disclosure constructs a pipeline that follows two steps: a 2D (two-dimensional) skeleton estimator in multiple camera views, and a 2D to 3D pose upscaling network that can output an estimated 3D pose. The skeleton estimator can estimate the two-dimensional coordinates or two-dimensional heat map of each feature point that constitutes the skeleton model of the target.

[0049] For example, the present disclosure also utilizes multi-view consistency constraints to better converge the captured 2D pose to the 3D pose. For incomplete or erroneous 2D detection situations, the multi-view setting also solves the technical problem of depth ambiguity.

[0050] For example, the present disclosure adopts the confidence of 2D joints to integrate losses from different views to alleviate the impact of noise caused by the self-occlusion problem.

[0051] In view of the above technical problems, the present disclosure proposes a self-supervised training method in a multi-view training setting. By using a self-supervised method instead of explicit 3D ground truth annotations, only 2D pose estimates generated by a 2D pose estimator are needed as input to complete the training.

[0052] This not only overcomes the depth ambiguity problem through a multi-view setting, but also solves the problem of incomplete or erroneous 2D detection by leveraging information from other views.

[0053] For example, since information from different perspectives and monocular settings have a wider range of application scenarios, multi-view settings are more advantageous when training models. The present disclosure can apply multi-view settings only during training and monocular settings during testing, thereby combining the advantages of both paradigms.

[0054] For example, the structure of the present disclosure mixes the outputs of two branches of a neural network. The first branch takes a single image (from one camera view) as input and generates a 3D pose in 3D space; the second branch inputs multiple images from different camera views and outputs an estimated 3D pose. The method provided by the present disclosure allows all estimated 3D poses from both branches to be projected to any camera view.

[0055] In some embodiments, the method provided by the present disclosure may include two stages. The first stage uses a 2D pose estimator to estimate the 2D human pose; the second stage promotes these 2D estimation results to 3D space; combines the camera's geometric information, the 3D pose from the second branch, and the rotation matrix of the view from the first branch, and obtains the desired 3D pose by rotating in the first camera coordinate system.

[0056] That is, all 3D poses in the real-world coordinate system should be exactly the same, so each acquired pose can be projected back into each camera's own 3D camera coordinate system. In addition, the 3D pose is reprojected to each 2D camera view through the camera matrix, and a reprojection loss can be defined for each 3D to 2D projection.

[0057] In some embodiments, the present disclosure proposes a two-branch self-supervised method in a multi-view training setting to train a 2D to 3D upscaling network without 3D annotation of the data. The technical solution of the present disclosure relies only on geometric knowledge to construct a supervisory signal, thereby obtaining better generalization ability.

[0058] In some embodiments, the present disclosure proposes a cyclic view training scheme that is effective in utilizing multi-view consistency information and constraining the estimated 3D pose during training. In this way, the depth ambiguity problem is overcome, and information from other views can be used to handle incomplete or erroneous two-dimensional detection situations. In addition, the present disclosure utilizes the 2D joint confidence of different camera views to alleviate the self-occlusion problem.

[0059] In some embodiments, the present disclosure proposes a volumetric 3DPS network based on a voxel-based approach and 3DPS. The network is capable of exploring a sufficient state space, in which a volume cube is generated by grid sampling; then, the cube is sent to a voxel-based neural network to obtain a 3D pose.

[0060] The inventor of the present disclosure has found that the above-mentioned related technologies have the following problems: the machine learning model requires a large number of labeled data samples, resulting in a high cost of posture estimation. In view of this, the present disclosure proposes a training technical solution for the machine learning model. For example, the technical solution of the present disclosure can be implemented through the following embodiments.

[0061] Figure 1 A flowchart of a method for training a machine learning model according to some embodiments of the present disclosure is shown.

[0062] like Figure 1 As shown, in step 110, a plurality of two-dimensional images and a machine learning model to be trained are obtained. The plurality of two-dimensional images contain targets. The machine learning model includes a first three-dimensional posture estimation module and a second three-dimensional posture estimation module. For example, the target includes a human target, and the plurality of feature points of the human target include a plurality of joint points.

[0063] In some embodiments, the machine learning model includes a first three-dimensional posture estimation module and a second three-dimensional posture estimation module. The first three-dimensional posture estimation module includes a first two-dimensional detection module and a second dimension-raising network module; the second three-dimensional posture estimation module includes a second two-dimensional detection module and a three-dimensional 3DPS module. The first two-dimensional detection module and the second two-dimensional detection module may be the same or different.

[0064] In some embodiments, when there are 4 camera views, there are 4 synchronized camera views and camera projection matrices, and all cameras capture 2D images of the same object and the same scene.

[0065] For example, multiple two-dimensional images processed by a machine learning model are acquired based on the viewing angles and projection matrices of multiple synchronized shooting devices to ensure that the multiple two-dimensional images remain synchronized in time, thereby improving the training effect and processing accuracy of the machine learning model.

[0066] In step 120, a first three-dimensional posture estimation module is used to estimate a first three-dimensional posture of the target according to any two-dimensional image. For example, the two-dimensional image is acquired from a first shooting angle by a monocular camera.

[0067] In this way, the training of the first 3D pose estimation module can be completed without labeling the training samples. The trained first 3D pose estimation module can accurately estimate the 3D pose of the target in a monocular camera scene without setting up multiple cameras, thereby reducing costs.

[0068] In some embodiments, the first three-dimensional pose may include at least one of a three-dimensional heat map or three-dimensional coordinates.

[0069] For example, when the trained first 3D posture estimation module is used for estimation, the first 3D posture in the form of 3D coordinates can be output to the user, thereby providing the user with more accurate information. The user can perform subsequent operations such as behavior recognition and action warning based on the 3D coordinates.

[0070] For example, in the process of training the first 3D posture estimation module, the first 3D posture in the form of a 3D heat map can be output. The heat map can represent the range of feature points instead of a fixed point, providing the system with a higher fault tolerance rate, thereby improving the training effect of the first 3D posture estimation module.

[0071] In some embodiments, based on any two-dimensional image, a first two-dimensional detection module is used to estimate the two-dimensional posture of the target.

[0072] For example, a two-dimensional pose estimator is used to predict a two-dimensional pose. For each frame of a two-dimensional image, the two-dimensional image can be cropped according to the bounding box of the target in the two-dimensional image detected by the two-dimensional detector.

[0073] For example, the cropped 2D image from the perspective of the first camera among the 4 cameras is represented by I, and the cropped 2D images from the perspectives of the other 3 cameras are represented by Representation: Each cropped two-dimensional image is input into a two-dimensional posture estimator to estimate the two-dimensional posture X of the target.

[0074] For example, the backbone of a 2D pose estimator is represented by f and the weight parameter is θ. A 2D pose estimator can consist of two parts: a global network for roughly predicting the 2D pose and a fine network for refining the joints.

[0075] In some embodiments, a two-dimensional feature point heat map of any first two-dimensional image is estimated based on any first two-dimensional image, and a two-dimensional posture is estimated based on the two-dimensional feature point heat map, for example, by performing soft-argmax processing on the two-dimensional feature point heat map to estimate the two-dimensional posture.

[0076] For example, the heatmap H of I can be estimated by a 2D pose estimator:

[0077] H=f(I;θ)

[0078] For example, the two-dimensional pose X of the target can be estimated by H. The argmax algorithm can be used to estimate the two-dimensional pose X. The argmax algorithm can obtain the joint with the highest probability by calculating the maximum value point of each heat map.

[0079] However, the argmax algorithm may cut off the gradient flow, making it impossible to train the 2D pose estimator. To address this technical problem, for example, soft-argmax can be used to apply to the heat map H instead of argmax:

[0080]

[0081] H k is the heat map of the kth joint of the target captured by the first camera, and Ω is the domain of the heat map. The soft-argmax uses the probability G corresponding to position i k (i) Weight position i to obtain the weighted coordinates of position i; add all the weighted coordinates to obtain the two-dimensional coordinates of each joint as the two-dimensional posture of the target.

[0082] In the above embodiment, the soft-argmax process can ensure that the gradient flow is transferred from the output three-dimensional posture to the input two-dimensional image, so that the machine learning model will not cut off the gradient flow, thereby improving the accuracy of posture estimation.

[0083] In some embodiments, the first three-dimensional posture is estimated using a second dimension-raising network module based on the two-dimensional posture of the target. For example, the N detected 2D joints are represented as Use 3D upscaling network to predict 3D pose

[0084] In some embodiments, the second dimension-raising network module includes a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer.

[0085] For example, the dimension-raising network w vThe goal is to estimate the joint position of the target in 3D space given only 2D input. The dimension-raising network may include a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer through a residual connection. The input layer of the dimension-raising network may input the coordinates of N (a positive integer such as 17) human joints and apply them to a fully connected layer with 1024 output channels; then, it is processed using a residual connection layer consisting of 4 modules connected through residual connections, each module may consist of two fully connected layers; then, it is processed using a batch normalization layer, a rectified linear unit, and a dropout layer; finally, the final feature output by a residual module is input into a linear layer to obtain the 3D posture Y:

[0086] Y=w v (X)

[0087] In the above embodiment, a 2D pose estimation network is applied to predict the 2D poses and heat maps of multiple input frame images; then, the confidence of all 2D feature points is upgraded to the confidence of 3D feature points to integrate the losses from different perspectives. In this way, the influence of noise caused by self-occlusion problem can be reduced, thereby improving the accuracy of pose estimation.

[0088] In step 130, a second three-dimensional posture estimation module is used to estimate a second three-dimensional posture of the target based on at least two two-dimensional images, wherein the viewing angles of the at least two two-dimensional images are different from each other.

[0089] For example, the second three-dimensional posture may include at least one of a three-dimensional heat map of the target or three-dimensional coordinates of each feature point of the target.

[0090] In some embodiments, the at least two two-dimensional images can be acquired from different shooting angles by a monocular camera, or by a multi-camera. For example, the two two-dimensional images can be acquired from the second angle and the third angle by a monocular camera, or by one of the binocular cameras from the second angle and the other from the third angle. The second angle and the third angle are different.

[0091] In some embodiments, the target includes multiple feature points. According to at least two two-dimensional images, the information of the three-dimensional space range where each of the multiple feature points of the target is located is estimated; according to the information of the three-dimensional space range, the second three-dimensional posture is estimated. For example, the three-dimensional space range includes a cube space range, a sphere space range, etc.

[0092] In some embodiments, the three-dimensional space range corresponding to the feature point may be a space range centered on the feature point. For example, a 3DPS network module may be used to estimate a three-dimensional heat map of the location of the feature point, and the three-dimensional space range corresponding to the feature point may be determined based on the three-dimensional heat map.

[0093] For example, the size of the three-dimensional space range can be determined by a preset threshold. The heat threshold can be used to determine the range of the three-dimensional heat map where the feature point is located and the heat is greater than the heat threshold as the three-dimensional space range of the feature point.

[0094] In some embodiments, a two-dimensional feature point heat map is estimated for each of at least two two-dimensional images, and information about a three-dimensional space range is estimated based on the two-dimensional feature point heat map, for example, using a 3DPS module to estimate information about a three-dimensional space range.

[0095] For example, receiving input 2D images from 3 different camera perspectives And output a 2D heat map for each 2D image of the viewing angle

[0096]

[0097] In some embodiments, the 3DPS module is a multi-view 3D pose prediction method that can explore sufficient state space and generate 3D human body part candidates through grid sampling; then, predict the 2D pose through a 2D detector; and generate the 3D pose using 2D priors and maximum likelihood estimation.

[0098] In some embodiments, in order to improve the accuracy and robustness of 3DPS, a volumetric algorithm and 3DPS can be combined to obtain a volumetric 3DPS network to estimate the three-dimensional pose.

[0099] For example, using a volumetric 3DPS network that receives a 2D heatmap as input Output 3D pose after processing

[0100] For example, the 3DPS module can be used to sample the 3D cubes around the joints of the human body instead of sampling the 3D joints. Apply the 3DPS method to the 2D heat map corresponding to the joint k under the viewing angle c. After processing, the volume cube corresponding to joint k under viewing angle c is obtained as the cube space range:

[0101]

[0102] is a volume cube generated according to the kth joint heat map of the cth camera view, and T represents the processing of the 3DPS method. For example, the size of the volume cube can be 32×32×32.

[0103] The volume cubes at each perspective can be added together to get the sum of the target volume cubes

[0104]

[0105] Represents the sum of the volume cubes of joint k under multiple viewpoints c.

[0106] In some embodiments, a three-dimensional feature point heat map is estimated based on information of the three-dimensional spatial range; and a second three-dimensional posture is estimated based on the three-dimensional feature point heat map.

[0107] For example, the sum of the volume cubes is fed into a learnable volume convolutional neural network p q The architecture of the volume convolutional neural network can be similar to the voxel-to-voxel network architecture. The three-dimensional feature point heat map H can be obtained. 3D :

[0108] H 3D =p q (V sum )

[0109] V sum is the sum of the volume cubes of each joint k The splicing result of , q represents the weight of the voxel-to-voxel network.

[0110] In some embodiments, the second three-dimensional pose is estimated by performing a soft-argmax process on the three-dimensional feature point heat map.

[0111] For example, applying soft-argmax to the heatmap H 3D , get the three-dimensional posture

[0112]

[0113] represents the heat map of the kth joint of the estimated target in three-dimensional space, Ω represents the domain; thus, the position of the kth three-dimensional joint can be obtained As a 3D pose:

[0114]

[0115] In the above embodiment, the volume algorithm (i.e., estimating the three-dimensional space range corresponding to the feature point) and 3DPS are combined to sample the three-dimensional space range around the feature point of the target, and send the three-dimensional space range to the voxel-based neural network to obtain the three-dimensional pose. In this way, the accuracy and robustness of 3DPS are improved, thereby improving the accuracy of pose estimation.

[0116] In step 140, at least one of the first 3D pose estimation module or the second 3D pose estimation module is trained according to the difference between the first 3D pose and the second 3D pose.

[0117] In the above embodiments, the machine learning model of the present invention is trained in a multi-view setting using a self-supervised approach, so that the machine learning model benefits from information from different viewpoints, thereby improving the accuracy of posture estimation.

[0118] However, since the training data does not have the annotation of the true value of the 3D posture, it is difficult for the 3D posture generated from the two branches to be located at the same absolute 3D position, which may cause the training to be difficult to converge. In view of the above technical problems, a pre-training scheme can be applied to help locate the 3D posture to ensure that the 3D posture generated by the two branches is located at the same absolute 3D position. For example, the above technical scheme can be implemented through the following embodiments.

[0119] In some embodiments, a first key point of a first three-dimensional posture and a second key point of a second three-dimensional posture are obtained; based on the first key point and the second key point, the perspective of the second three-dimensional posture and the perspective of the first three-dimensional posture are corrected to a consistent perspective; based on the difference between the corrected first three-dimensional posture and the second three-dimensional posture, at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module is trained.

[0120] In some embodiments, the first key point and the second key point are estimated using the first dimension-raising network module. For example, the first key point is located at the center of the first three-dimensional posture, and the second key point is located at the center of the second three-dimensional posture. For example, the pelvis of the human body can be used as the key point at the center. For example, the first key point and the second key point can be estimated based on each feature point.

[0121] For example, a first dimension-raising network different from a second dimension-raising network module can be used to predict the center position of a three-dimensional posture. The center first dimension-raising network module and the second dimension-raising network module can have the same network structure but different weight parameters. The first dimension-raising network module can include u γ and Two parts, u γ Used to predict the first key point Y under the first camera perspective pelvis , Used to predict the second keypoints from randomly selected views from the second, third, and fourth camera views

[0122] Y pelvis =u γ (X)

[0123]

[0124] is the predicted 2D pose at a randomly chosen viewpoint. The predicted keypoints can then be used to localize the 3D pose, and the 3D pose can be moved to the predicted position during training to ensure that the viewpoint of the second 3D pose is corrected to the same viewpoint as the first 3D pose.

[0125] In some embodiments, the perspective of the second three-dimensional pose is corrected to the perspective of the first three-dimensional pose to generate a third three-dimensional pose, so as to train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the third three-dimensional pose and the first three-dimensional pose.

[0126] For example, the correspondence between the feature points on the second 3D posture and the first 3D posture can be determined; single-line alignment is performed based on the first key point and the second key point, and the positions of the feature points on the edge of the second 3D posture and the edge of the first 3D posture that have a corresponding relationship. In this way, the perspective of the second 3D posture can be guaranteed to be consistent with the perspective of the first 3D posture, thereby improving the accuracy of posture estimation.

[0127] In some embodiments, the MSE loss and the two-dimensional reprojection loss are determined based on the difference between the first three-dimensional pose and the second three-dimensional pose; and at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained based on the weighted mean of the MSE loss and the two-dimensional reprojection loss until the training end condition is met.

[0128] For example, two different loss functions, MSE loss for 3D and reprojection loss for 2D, can be applied to train a machine learning model. The reprojection loss can further constrain the consistency of multiple views, thereby improving the performance of the machine learning model.

[0129] This is because, in the absence of depth information, a 2D detector corresponds to multiple possibilities of 3D poses in different orientations, and 2D reprojection can help constrain it to the correct orientation by leveraging information from other viewpoints, thereby improving the performance of machine learning models.

[0130] For example, the MSE loss can be:

[0131]

[0132] Y k is the first 3D posture of feature point k, is the second 3D posture of feature point k.

[0133] For example, the 2D reprojection loss is:

[0134]

[0135]

[0136] P c =[p1p2p3] T .

[0137] p1p2p3 represents the coordinate transformation information required for coordinate transformation.

[0138] In some embodiments, the total loss for training the machine learning model is:

[0139]

[0140] α is a weight coefficient set as needed to adjust the importance of the two losses in training.

[0141] Figure 2 A schematic diagram showing a method for training a machine learning model according to some other embodiments of the present disclosure.

[0142] like Figure 2 As shown, in step 210, according to the training method in any of the above embodiments, the first three-dimensional posture estimation module of the machine learning model is trained.

[0143] In step 220, a two-dimensional image containing the target is acquired; and based on the two-dimensional image, the three-dimensional pose of the target is estimated using the trained first three-dimensional pose estimation module.

[0144] In some embodiments, usage requirements of the machine learning model are obtained; and the machine learning model is converted into a format that matches the usage requirements.

[0145] In some embodiments, in response to a user's setting operation, a deployment environment and model parameters are set for the machine learning model.

[0146] In some embodiments, a three-dimensional gesture is displayed.

[0147] In some embodiments, the target includes multiple feature points, and information about the three-dimensional space range where each of the multiple feature points of the target is located is estimated based on at least two two-dimensional images; and a second three-dimensional posture is estimated based on the information about the three-dimensional space range.

[0148] In some embodiments, a two-dimensional feature point heat map is estimated for each of at least two two-dimensional images; and information about a three-dimensional spatial range is estimated based on the two-dimensional feature point heat map.

[0149] In some embodiments, a three-dimensional feature point heat map is estimated based on information of the three-dimensional spatial range; and a second three-dimensional posture is estimated based on the three-dimensional feature point heat map.

[0150] In some embodiments, the second three-dimensional pose is estimated by performing a soft-argmax process on the three-dimensional feature point heat map.

[0151] In some embodiments, the three-dimensional spatial extent includes a cubic spatial extent.

[0152] In some embodiments, the 3DPS module is used to estimate the three-dimensional spatial range information.

[0153] In some embodiments, a first key point of a first three-dimensional posture and a second key point of a second three-dimensional posture are obtained; based on the first key point and the second key point, the perspective of the second three-dimensional posture and the perspective of the first three-dimensional posture are corrected to a consistent perspective; based on the difference between the corrected first three-dimensional posture and the second three-dimensional posture, at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module is trained.

[0154] In some embodiments, the perspective of the second three-dimensional pose is corrected to the perspective of the first three-dimensional pose to generate a third three-dimensional pose, so as to train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the third three-dimensional pose and the first three-dimensional pose.

[0155] In some embodiments, the first key point and the second key point are estimated using a first dimension raising network module.

[0156] In some embodiments, the first key point is located at a center position of the first three-dimensional pose, and the second key point is located at a center position of the second three-dimensional pose.

[0157] In some embodiments, based on any two-dimensional image, a first two-dimensional detection module is used to estimate the two-dimensional posture of the target; based on the two-dimensional posture of the target, a second dimensionality-raising network module is used to estimate the first three-dimensional posture.

[0158] In some embodiments, based on any first two-dimensional image, a two-dimensional feature point heat map of any first two-dimensional image is estimated; based on the two-dimensional feature point heat map, a two-dimensional posture is estimated.

[0159] In some embodiments, the 2D pose is estimated by performing a soft-argmax process on the 2D feature point heat map.

[0160] In some embodiments, the second dimension-raising network module includes a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer.

[0161] In some embodiments, the MSE loss and the two-dimensional reprojection loss are determined based on the difference between the first three-dimensional pose and the second three-dimensional pose; and at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained based on the weighted mean of the MSE loss and the two-dimensional reprojection loss until the training end condition is met.

[0162] In some embodiments, the target includes a human target, and the plurality of feature points of the human target includes a plurality of joint points.

[0163] In some embodiments, at least one of the first three-dimensional gesture or the second three-dimensional gesture is displayed.

[0164] Figure 3 A schematic diagram showing a method for training a machine learning model according to some embodiments of the present disclosure.

[0165] like Figure 3 As shown in the figure, the machine learning model consists of two branches. The first branch inputs a single 2D image of the first camera perspective (from one camera perspective) and generates a first 3D pose in 3D space. The second branch inputs three 2D images of the other three camera perspectives and outputs an estimated second 3D pose. The second 3D pose can be rotated to the first camera perspective to achieve multi-perspective consistency loss. In this way, the 3D poses estimated from different perspectives have the same skeleton before geometric transformation. The predicted 3D pose can be reprojected to each camera perspective to obtain additional reprojection errors.

[0166] In some embodiments, the machine learning model includes Figure 3 The two-dimensional detector, dimension-raising network and volume 3DPS network in the embodiment. The two-dimensional detector includes the first two-dimensional detection module and the second two-dimensional detection module in any embodiment; the dimension-raising network includes the second dimension-raising network module in any embodiment; and the volume 3DPS network includes the 3DPS module in any embodiment.

[0167] For example, the first 3D pose estimation module includes Figure 3 The two-dimensional detector and dimension-raising network in the second three-dimensional posture estimation module include Figure 3 2D detector and volumetric 3DPS network in.

[0168] In some embodiments, a 2D detector is applied to predict a 2D pose and a 2D heat map from four input 2D image frames; then, a dimensionality-enhancing network is used to enhance the confidence of each 2D feature point to 3D.

[0169] Figure 3 The network structure for 4 camera views is shown. There are 4 synchronized camera views and camera projection matrices, and all cameras capture the same object and the same scene. First, a 2D detector is used to obtain the 2D pose; for each frame of the 2D image, the bounding box of the object detected by the 2D detector is used to crop the 2D image.

[0170] For example, the cropped 2D image from the perspective of the first camera among the 4 cameras is represented by I, and the cropped 2D images from the perspectives of the other 3 cameras are represented by 1,2,3}; each cropped 2D image is input into the 2D pose estimator to estimate the 2D pose X of the target. For example, the backbone of the 2D pose estimator is represented by f, and the weight parameter is θ. The 2D pose estimator can be composed of two parts: a global network for roughly predicting the 2D pose; and a fine network for refining the joints.

[0171] like Figure 3 As shown in Figure 2, the training process has two branches. In the first branch, Represented as N detected 2D joints; then, a 3D upscaling network is used to predict the 3D pose In the other branch, it receives the input 2D image from the 3 camera views and outputs a 2D heat map for each view. Then, using the volumetric 3DPS network, it takes in the heatmap as input and outputs a 3D pose. The heatmap can be estimated by a 2D detector:

[0172] H=f(I;θ),

[0173]

[0174] The 2D pose X for the first branch is estimated by H. The argmax algorithm can be used to estimate the 2D pose X. The argmax algorithm can obtain the joint with the highest probability by calculating the maximum value point of each heat map.

[0175] However, the argmax algorithm may cut off the gradient flow, making it impossible to train the 2D pose estimator. To address this technical problem, for example, soft-argmax can be used to apply to the heat map H instead of argmax:

[0176]

[0177] In some embodiments, the three-dimensional pose can be estimated with zero as the center, and Y and The value of is the 3D position relative to the fixed root joint. The predicted 3D pose is in its own pose coordinate system during training, and can then be The 3D loss is obtained by rotating to the first camera coordinates; the reprojection loss of the pose at each viewpoint is obtained by reprojecting the global 3D pose into the second external space.

[0178] For example, the Lifting Network vThe goal is to estimate the joint position of the target in 3D space given only 2D input. The dimension-raising network may include a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer through a residual connection. The input layer of the dimension-raising network may input the coordinates of N (a positive integer such as 17) human joints and apply them to a fully connected layer with 1024 output channels; then, it is processed using a residual connection layer consisting of 4 modules connected through residual connections, each module may consist of two fully connected layers; then, it is processed using a batch normalization layer, a rectified linear unit, and a dropout layer; finally, the final feature output by a residual module is input into a linear layer to obtain the 3D posture Y:

[0179] Y=w v (X)

[0180] The 3DPS module is a multi-view 3D pose prediction method that can explore sufficient state space and generate 3D human part candidates through grid sampling; then, it predicts the 2D pose through a 2D detector; and generates the 3D pose using 2D priors and maximum likelihood estimation.

[0181] In order to improve the accuracy and robustness of 3DPS, the volumetric algorithm and 3DPS can be combined to obtain a volumetric 3DPS network to estimate the 3D posture. The 3DPS module can be used to sample the 3D cubes around the joints of the human body instead of sampling the 3D joints. The 3DPS method is applied to the 2D heat map corresponding to the joint k under the view angle c. After processing, the volume cube corresponding to joint k under viewing angle c is obtained as the cube space range:

[0182]

[0183] is a volume cube generated according to the kth joint heat map of the cth camera view, and T represents the processing of the 3DPS method. For example, the size of the volume cube can be 32×32×32.

[0184] The volume cubes at each perspective can be added together to get the sum of the target volume cubes

[0185]

[0186] Represents the sum of the volume cubes of joint k under multiple viewpoints c.

[0187] The sum of the volume cubes is input into the learnable volume convolutional neural network p q The architecture of the volume convolutional neural network can be similar to the voxel-to-voxel network architecture. The three-dimensional feature point heat map H can be obtained.3D :

[0188] H 3D =p q (V sum )

[0189] V sum is the sum of the volume cubes of each joint k The splicing result of , q represents the weight of the voxel-to-voxel network.

[0190] Apply soft-argmax to the heatmap H 3D , get the three-dimensional posture

[0191]

[0192] represents the heat map of the kth joint of the estimated target in three-dimensional space, Ω represents the domain; thus, the position of the kth three-dimensional joint can be obtained As a 3D pose:

[0193]

[0194] For example, using the L2 norm between the 3D poses predicted from different views It ensures consistency across multiple views. If the prediction result is accurate, the loss should be zero. This is because the same object captured from different camera views should have the same absolute 3D pose in 3D space.

[0195] However, this solution does not work well for more than two views. During training, the L2-norm loss of 2D1’s view vs. 2D2’s view conflicts with the L2-norm loss of 2D2’s view vs. 3’s view. This is because the up-dimensionality network learns a 3D pose that is only applicable to one view without the depth information from the camera view.

[0196] To address the above technical issues, the 3D poses generated by different camera perspectives can be converted into the same 3D pose. For example, a cyclic perspective training scheme can be used. Figure 3 The machine learning model in consists of two branches, the first branch is a camera perspective, followed by a dimensionality-increasing network; the second branch is three camera perspectives, followed by a volumetric 3DPS network. The input 2D images of these four perspectives can be randomly swapped, so that all possible 3D perspective combinations can be trained to solve the multi-perspective consistency problem.

[0197] For example, after a round of training using the training method in any of the above embodiments, one two-dimensional image can be selected from at least two two-dimensional images in the previous round of training as the first two-dimensional image, the remaining one of the at least two two-dimensional images can be determined as the second two-dimensional image, and any two-dimensional image in the previous round of training can be determined as the second two-dimensional image; based on the first two-dimensional image, the first three-dimensional posture estimation module is used to estimate the first three-dimensional posture of the target in this round of training; based on the second two-dimensional image, the second three-dimensional posture estimation module is used to estimate the second three-dimensional posture of the target in this round of training; based on the difference between the first three-dimensional posture in this round of training and the second three-dimensional posture in this round of training, at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module is trained.

[0198] In the above embodiments, the machine learning model of the present invention is trained in a multi-view setting using a self-supervised approach, so that the machine learning model benefits from information from different viewpoints, thereby improving the accuracy of posture estimation.

[0199] However, since the training data does not have the annotation of the true value of the 3D pose, the 3D poses generated from the two branches are difficult to be located at the same absolute 3D position, which may cause the training to be difficult to converge. To address the above technical issues, a pre-training scheme can be applied to help locate the 3D pose to ensure that the 3D poses generated by the two branches are located at the same absolute 3D position.

[0200] For example, a first dimension-raising network different from a second dimension-raising network module can be used to predict the center position of a three-dimensional posture. The center first dimension-raising network module and the second dimension-raising network module can have the same network structure but different weight parameters. The first dimension-raising network module can include u γ and Two parts, u γ Used to predict the first key point Y under the first camera perspective pelvis , Used to predict the second keypoints from randomly selected views from the second, third, and fourth camera views

[0201] Y pelvis =u γ (X)

[0202]

[0203] The predicted key points can then be used to localize the 3D pose, and the 3D pose can be moved to the predicted position during training to ensure that the perspective of the second 3D pose is corrected to a consistent perspective with the perspective of the first 3D pose.

[0204] For example, Figure 3The double arrow between the first three-dimensional pose and the third three-dimensional pose may indicate that the machine learning model is trained by comparing the difference between the first three-dimensional pose and the third three-dimensional pose.

[0205] For example, the 3D heat map generated by the dimensionality-raising network can also be processed using soft-argmax to obtain the first 3D pose. The soft-argmax process can ensure that the gradient flow is passed from the output 3D pose to the input 2D image, so that the machine learning model does not cut off the gradient flow.

[0206] Two different loss functions, 3D MSE loss and 2D reprojection loss, can be applied to train machine learning models. Reprojection loss can further constrain the consistency of multiple views, thereby improving the performance of machine learning models.

[0207] This is because, in the absence of depth information, a 2D detector corresponds to multiple possibilities of 3D poses in different orientations, and 2D reprojection can help constrain it to the correct orientation by leveraging information from other viewpoints, thereby improving the performance of machine learning models.

[0208] For example, the MSE loss can be:

[0209]

[0210] Y k is the first 3D posture of feature point k, is the second 3D posture of feature point k.

[0211] For example, the 2D reprojection loss is:

[0212]

[0213]

[0214] P c =[p1p2p3] T .

[0215] p1p2p3 represents the coordinate transformation information required for coordinate transformation.

[0216] In some embodiments, the total loss for training the machine learning model is:

[0217]

[0218] α is a weight coefficient set as needed to adjust the importance of the two losses in training.

[0219] For example, multi-view 3D human pose estimation methods can use input 2D images from multiple different viewpoints, thus achieving better performance than using a single monocular camera view.

[0220] In some embodiments, during the training phase, a single view process and a voxel V2V 3DPS (i.e., volume 3DPS) neural network may be used to assist in the retraining of the single view process. The volume 3DPS network includes a 3DPS module and a V2V network for estimating the cube in which the feature points are located.

[0221] For example, a multi-view setting can be used only in the training phase, and a single-view setting is required in the testing phase. In this way, the training of the 3D pose estimation model can be realized without the need for 3D data annotation.

[0222] For example, reprojection loss helps constrain multi-view information during training. Through the voxel V2V 3DPS neural network and the training process based on cyclic perspective, technical problems caused by data labeling can be avoided, and an unlabeled training process can be achieved, thereby reducing training costs and thresholds.

[0223] Figure 4 A schematic diagram showing a method for estimating a three-dimensional posture according to some embodiments of the present disclosure.

[0224] like Figure 4 As shown in Figure 1, the machine learning model inputs each frame of the input 2D image into the detection box detector to obtain the cropped 2D images and inputs them into the backbone network of the 2D pose estimator; then, the estimated 2D pose is input into the dimension-increasing network to obtain the estimated 3D pose. For example, the multi-view setting is only applicable to the training phase, and the monocular camera setting is followed in the testing phase.

[0225] Figures 5a to 5e A schematic diagram of a machine learning model development and deployment platform according to some embodiments of the present disclosure is shown.

[0226] like Figure 5a As shown, upload multi-view pedestrian data graphs (i.e., two-dimensional images) on the data upload page, and make the time sequence correspondence between each pedestrian data graph according to the corresponding naming rules. When uploading data, in addition to the time sequence information transmitted by the camera itself, no manual data annotation is required to support the subsequent training process and test deployment.

[0227] For example, the data upload page includes a task details area and a data file list area. The task details area may include relevant information about the training task, and the data file list area may include relevant information about the uploaded data.

[0228] like Figure 5bAs shown in the figure, on the model building page, upload and write the code of the machine learning model to be trained through the notebook. The uploaded code can include the settings of the training model and the test model and the code of related processes.

[0229] For example, the model upload and writing area of ​​the model building page includes information about the notebook, information about the development environment, and so on.

[0230] like Figure 5c As shown in the figure, on the model training page, the corresponding parameters of the model are designed and adjusted accordingly. When the model is trained, a multi-branch perspective is needed to assist the training process, so that the training itself can achieve better results, and no training annotation is required.

[0231] For example, the model configuration area of ​​the model training page includes configuration functions for training tasks, configuration functions for lithium acid resources, etc.

[0232] like Figure 5d As shown in the figure, on the model conversion page, if the user needs to convert the model, the trained model can be converted to facilitate subsequent deployment. For example, the machine learning model can be converted from pytorch to a model with an architecture such as ONNX. When deploying the model, different customer needs can be met through model conversion and precision correction.

[0233] For example, the Conversion Configuration area of ​​the Model Conversion page includes information about the machine learning model and the conversion configuration capabilities.

[0234] like Figure 5e As shown in the figure, on the model deployment page, select the trained model and set the relevant deployment environment and related parameters to ensure that the model can run correctly when it is called.

[0235] For example, a machine learning model can use HTTP (HyperText Transfer Protocol) mode. The test input to the model can be a single image of a pedestrian from a single perspective. The trained model can recognize the 3D posture of the pedestrian in the current image, thus providing a basis for subsequent action recognition or behavior analysis.

[0236] During the deployment phase, this model only needs to use the single-process weight model in the test file to adapt to the needs of the real environment and reduce the hardware cost and conditions required for operation.

[0237] For example, the Model Deployment area of ​​the Model Deployment page includes information about the model and information about the service to be processed.

[0238] Figure 6a A block diagram of a training apparatus for a machine learning model according to some embodiments of the present disclosure is shown.

[0239] like Figure 6a As shown, the training device 6a of the machine learning model includes at least one processor 61a, and the at least one processor 61a is configured to: obtain multiple two-dimensional images and a machine learning model to be trained, the multiple two-dimensional images contain targets, and the machine learning model includes a first three-dimensional posture estimation module and a second three-dimensional posture estimation module; based on any two-dimensional image, using the first three-dimensional posture estimation module, estimate the first three-dimensional posture of the target; based on at least two two-dimensional images, using the second three-dimensional posture estimation module to estimate the second three-dimensional posture of the target, and the viewing angles of at least two two-dimensional images are different; based on the difference between the first three-dimensional posture and the second three-dimensional posture, train at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module.

[0240] In some embodiments, the target includes multiple feature points, and information about the three-dimensional space range where each of the multiple feature points of the target is located is estimated based on at least two two-dimensional images; and a second three-dimensional posture is estimated based on the information about the three-dimensional space range.

[0241] In some embodiments, a two-dimensional feature point heat map is estimated for each of at least two two-dimensional images; and information about a three-dimensional spatial range is estimated based on the two-dimensional feature point heat map.

[0242] In some embodiments, a three-dimensional feature point heat map is estimated based on information of the three-dimensional spatial range; and a second three-dimensional posture is estimated based on the three-dimensional feature point heat map.

[0243] In some embodiments, the second three-dimensional pose is estimated by performing a soft-argmax process on the three-dimensional feature point heat map.

[0244] In some embodiments, the three-dimensional spatial extent includes a cubic spatial extent.

[0245] In some embodiments, the 3DPS module is used to estimate the three-dimensional spatial range information.

[0246] In some embodiments, a first key point of a first three-dimensional posture and a second key point of a second three-dimensional posture are obtained; based on the first key point and the second key point, the perspective of the second three-dimensional posture and the perspective of the first three-dimensional posture are corrected to a consistent perspective; based on the difference between the corrected first three-dimensional posture and the second three-dimensional posture, at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module is trained.

[0247] In some embodiments, the perspective of the second three-dimensional pose is corrected to the perspective of the first three-dimensional pose to generate a third three-dimensional pose, so as to train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module based on the difference between the third three-dimensional pose and the first three-dimensional pose.

[0248] In some embodiments, the first key point and the second key point are estimated using a first dimension raising network module.

[0249] In some embodiments, the first key point is located at a center position of the first three-dimensional pose, and the second key point is located at a center position of the second three-dimensional pose.

[0250] In some embodiments, based on any two-dimensional image, a first two-dimensional detection module is used to estimate the two-dimensional posture of the target; based on the two-dimensional posture of the target, a second dimensionality-raising network module is used to estimate the first three-dimensional posture.

[0251] In some embodiments, based on any first two-dimensional image, a two-dimensional feature point heat map of any first two-dimensional image is estimated; based on the two-dimensional feature point heat map, a two-dimensional posture is estimated.

[0252] In some embodiments, the 2D pose is estimated by performing a soft-argmax process on the 2D feature point heat map.

[0253] In some embodiments, the second dimension-raising network module includes a batch normalization layer, a dropout layer, a rectified linear layer, and a residual connection layer.

[0254] In some embodiments, the MSE loss and the two-dimensional reprojection loss are determined based on the difference between the first three-dimensional pose and the second three-dimensional pose; and at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained based on the weighted mean of the MSE loss and the two-dimensional reprojection loss until the training end condition is met.

[0255] In some embodiments, the target includes a human target, and the plurality of feature points of the human target includes a plurality of joint points.

[0256] In some embodiments, the training device 6a further includes: a display device 62a for displaying at least one of the first three-dimensional gesture or the second three-dimensional gesture.

[0257] Figure 6b A block diagram of a three-dimensional pose estimation device according to some embodiments of the present disclosure is shown.

[0258] like Figure 6b As shown, in some embodiments, a three-dimensional posture estimation device 6b includes: a training device 6a for a machine learning model in any of the above embodiments, configured to train a first three-dimensional posture estimation module of the machine learning model; and at least one processor 61b, wherein the at least one processor 61b is configured to: obtain a two-dimensional image containing a target; and estimate the three-dimensional posture of the target based on the two-dimensional image using the trained first three-dimensional posture estimation module.

[0259] In some embodiments, the processor 61b is configured to: obtain usage requirements of the machine learning model; and convert the machine learning model into a format that matches the usage requirements.

[0260] In some embodiments, the processor 61b is configured to: set the deployment environment and model parameters of the machine learning model in response to the user's setting operation.

[0261] In some embodiments, the display device 62a is used to display three-dimensional gestures.

[0262] Figure 7 A block diagram showing some embodiments of the electronic device of the present disclosure.

[0263] like Figure 7 As shown, the electronic device 7 of this embodiment includes: a memory 71 and a processor 72 coupled to the memory 71, and the processor 72 is configured to execute the training method of the machine learning model or the three-dimensional posture estimation method in any embodiment of the present disclosure based on the instructions stored in the memory 71.

[0264] The memory 71 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0265] Figure 8 A block diagram showing some other embodiments of the electronic device of the present disclosure.

[0266] like Figure 8 As shown, the electronic device 8 of this embodiment includes: a memory 810 and a processor 820 coupled to the memory 810, and the processor 820 is configured to execute the training method of the machine learning model or the three-dimensional posture estimation method in any one of the aforementioned embodiments based on the instructions stored in the memory 810.

[0267] The memory 810 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0268] The electronic device 8 may also include an input / output interface 830, a network interface 840, a storage interface 850, etc. These interfaces 830, 840, 850, the memory 810, and the processor 820 may be connected, for example, via a bus 860. The input / output interface 830 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, a touch screen, a microphone, and a speaker. The network interface 840 provides a connection interface for various networked devices. The storage interface 850 provides a connection interface for external storage devices such as an SD card and a USB flash drive.

[0269] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0270] So far, the present invention has been described in detail. In order to avoid obscuring the concept of the present invention, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.

[0271] The method and system of the present disclosure may be implemented in many ways. For example, the method and system of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0272] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A training device for a machine learning model, comprising at least one processor, wherein the at least one processor is configured to: Acquire a plurality of two-dimensional images and the machine learning model to be trained, wherein the plurality of two-dimensional images contain targets, and the machine learning model includes a first three-dimensional posture estimation module and a second three-dimensional posture estimation module; According to any one of the two-dimensional images, using the first three-dimensional posture estimation module, estimating a first three-dimensional posture of the target; estimating a second three-dimensional posture of the target using the second three-dimensional posture estimation module according to at least two two-dimensional images, wherein the viewing angles of the at least two two-dimensional images are different from each other; At least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained according to a difference between the first three-dimensional pose and the second three-dimensional pose.

2. The training device according to claim 1, wherein: The target includes a plurality of feature points, and estimating the second three-dimensional posture of the target by using the second three-dimensional posture estimation module according to at least two two-dimensional images includes: According to the at least two two-dimensional images, estimating information of a three-dimensional spatial range where each of the plurality of feature points of the target is located; The second three-dimensional posture is estimated according to the information of the three-dimensional space range.

3. The training device according to claim 2, wherein: The information of estimating the three-dimensional space range of each of the multiple characteristic nodes of the target according to the at least two two-dimensional images includes: estimating a two-dimensional feature point heat map for each of the at least two two-dimensional images; According to the two-dimensional feature point heat map, information of the three-dimensional space range is estimated.

4. The training device according to claim 2, wherein: The estimating the second three-dimensional posture according to the information of the three-dimensional space range includes: estimating a three-dimensional feature point heat map based on the information of the three-dimensional spatial range; The second three-dimensional posture is estimated according to the three-dimensional feature point heat map.

5. The training device according to claim 4, wherein: The estimating the second three-dimensional posture according to the three-dimensional feature point heat map includes: The second three-dimensional posture is estimated by performing soft-argmax processing on the three-dimensional feature point heat map.

6. The training device according to claim 2, wherein: The three-dimensional space range includes a cubic space range.

7. The training device according to claim 2, wherein: The information of estimating the three-dimensional space range of each of the multiple feature points of the target according to the at least two two-dimensional images includes: The three-dimensional image structure module is used to estimate the information of the three-dimensional space range.

8. The training device according to any one of claims 1 to 7, wherein: The training of at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module according to the difference between the first three-dimensional posture and the second three-dimensional posture comprises: Acquire a first key point of the first three-dimensional posture and a second key point of the second three-dimensional posture; Correcting the perspective of the second three-dimensional posture and the perspective of the first three-dimensional posture to be consistent with each other according to the first key point and the second key point; At least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained according to the difference between the corrected first three-dimensional pose and the second three-dimensional pose.

9. The training device according to claim 8, wherein: The step of correcting the perspective of the second three-dimensional posture and the perspective of the first three-dimensional posture to be consistent with each other according to the first key point and the second key point comprises: The perspective of the second three-dimensional pose is corrected to the perspective of the first three-dimensional pose to generate a third three-dimensional pose, so as to train at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module according to the difference between the third three-dimensional pose and the first three-dimensional pose.

10. The training device according to claim 8, wherein: The acquiring of the first key point of the first three-dimensional posture and the second key point of the second three-dimensional posture comprises: The first key point and the second key point are estimated by using a first dimension-raising network module.

11. The training device according to claim 8, wherein: The first key point is located at a center position of the first three-dimensional posture, and the second key point is located at a center position of the second three-dimensional posture.

12. The training device according to any one of claims 1 to 7, wherein: The estimating the first three-dimensional posture of the target by using the first three-dimensional posture estimation module according to any two-dimensional image comprises: According to any one of the two-dimensional images, using a first two-dimensional detection module to estimate the two-dimensional posture of the target; The first three-dimensional posture is estimated by using a second dimension-raising network module according to the two-dimensional posture of the target.

13. The training device according to claim 12, wherein: The estimating the two-dimensional posture of the target by using a first two-dimensional detection module according to any one of the first two-dimensional images comprises: According to any one of the first two-dimensional images, estimating a two-dimensional feature point heat map of any one of the first two-dimensional images; The two-dimensional posture is estimated according to the two-dimensional feature point heat map.

14. The training device according to claim 13, wherein: The estimating the two-dimensional posture according to the two-dimensional feature point heat map comprises: The two-dimensional posture is estimated by performing soft-argmax processing on the two-dimensional feature point heat map.

15. The training device according to claim 12, wherein: The second dimension-raising network module includes a batch normalization layer, a random inactivation dropout layer, a rectified linear layer and a residual connection layer.

16. The training device according to any one of claims 1 to 7, wherein: The training of at least one of the first three-dimensional posture estimation module or the second three-dimensional posture estimation module according to the difference between the first three-dimensional posture and the second three-dimensional posture comprises: determining a mean square error loss and a two-dimensional reprojection loss based on a difference between the first three-dimensional pose and the second three-dimensional pose; According to the weighted mean of the MSE loss and the two-dimensional reprojection loss, at least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained until a training end condition is met.

17. The training device according to any one of claims 1 to 7, wherein: The target includes a human target, and the multiple feature points of the human target include multiple joint points.

18. The training device according to any one of claims 1 to 7, further comprising: The display device is used to display at least one of the first three-dimensional gesture or the second three-dimensional gesture.

19. The estimation device according to any one of claims 1 to 7, wherein: The at least two two-dimensional images are acquired according to the viewing angles and projection matrices of a plurality of synchronized shooting devices.

20. A three-dimensional posture estimation device, comprising: A training device for a machine learning model according to any one of claims 1 to 19, configured to train a first three-dimensional pose estimation module of the machine learning model; as well as at least one processor, the at least one processor being configured to: Acquire a two-dimensional image containing the target; The three-dimensional posture of the target is estimated according to the two-dimensional image using a trained first three-dimensional posture estimation module.

21. The estimation device according to claim 20, wherein: The processor is configured to: Obtaining usage requirements for the machine learning model; Convert the machine learning model into a format that matches the usage requirements.

22. The estimation device according to claim 20, wherein: The processor is configured to: In response to the user's setting operation, the deployment environment and model parameters of the machine learning model are set.

23. The estimation device according to any one of claims 20 to 22, wherein: The display device is used to display the three-dimensional posture.

24. A method for training a machine learning model, comprising: According to any two-dimensional image containing the target, estimating a first three-dimensional posture of the target using a first three-dimensional posture estimation module of the machine learning model; estimating a second three-dimensional posture of the target using a second three-dimensional posture estimation module of the machine learning model according to at least two two-dimensional images containing the target, wherein the viewing angles of the plurality of second two-dimensional images are different from each other; At least one of the first three-dimensional pose estimation module or the second three-dimensional pose estimation module is trained according to a difference between the first three-dimensional pose and the second three-dimensional pose.

25. A method for estimating a three-dimensional posture, comprising: According to the training method of the machine learning model according to claim 24, training a first three-dimensional posture estimation module of the machine learning model; The three-dimensional posture of the target is estimated using the first three-dimensional posture estimation module according to the two-dimensional image containing the target.

26. An electronic device comprising: Memory; and A processor coupled to the memory, the processor being configured to execute the training method of the machine learning model described in claim 24 or the three-dimensional posture estimation method described in claim 25 based on instructions stored in the memory device.

27. A non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method of the machine learning model as described in claim 24 or the three-dimensional posture estimation method as described in claim 25.

Images