Model training method and apparatus, three-dimensional pose estimation method and apparatus, medium, and electronic device

Abstract

Provided are a model training method and apparatus, a three-dimensional pose estimation method and apparatus, and an electronic device, relating to the field of computer vision. The method comprises: acquiring at least one training sample, and inputting each training sample into an initial three-dimensional pose estimation model for training to obtain a trained three-dimensional pose estimation model. Each round of training comprises: inputting a sample image of the training sample into a current three-dimensional pose estimation model to obtain the three-dimensional Gaussian mixture representation of each key point on a preset three-dimensional coordinate system; for each key point, obtaining the two-dimensional Gaussian mixture representations of the key point on three mutually perpendicular two-dimensional planes in the preset three-dimensional coordinate system; for each key point, determining the loss value of the key point on each two-dimensional plane; and adjusting model parameters of the current three-dimensional pose estimation model on the basis of the loss value of each key point on each two-dimensional plane. The embodiments of the present application effectively improve the accuracy of three-dimensional pose estimation.

Description

Model training method, three-dimensional pose estimation method, device, medium and electronic device

[0001] The present application claims priority to the Chinese patent application No. 202410632464.8, filed on May 21, 2024, and entitled "Model training method, three-dimensional pose estimation method, device and electronic device", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of computer vision, in particular, the present application relates to a model training method, a three-dimensional pose estimation method, a device, a medium and an electronic device. BACKGROUND

[0003] Human pose estimation aims to predict the predefined key points on the human body, and is a basic task in computer vision. It is widely used in various vision tasks and is an important preprocessing operation for many downstream tasks such as human motion analysis, activity recognition, motion capture, etc.

[0004] In the related art, a pose estimation model can be used to estimate the pose of a to-be-identified picture, obtain coordinate information of each key point for describing the pose of an object, and then analyze the pose of the related object by using the obtained coordinate information.

[0005] In the process of training the pose estimation model, the directly-regressed three-dimensional coordinate values and the input image are not in the same space dimension, so the constraint on the coordinate values is an implicit and non-aligned constraint method. The pose estimation model cannot capture the internal information in the image well, and thus the performance obtained is poor. Furthermore, the coordinate information of the key points cannot be accurately obtained by using the trained pose estimation model, and the pose estimation effect is reduced. SUMMARY

[0006] The embodiments of the present application provide a model training method, a three-dimensional pose estimation method, a device, a medium and an electronic device, which can solve the above problems of the related art. The technical solution is as follows:

[0007] According to a first aspect of the embodiments of the present application, a model training method is provided, comprising:

[0008] Obtaining at least one training sample, each training sample comprising a sample image of a human body and annotation information, the annotation information comprising three-dimensional actual coordinates of at least one key point of the human body in the sample image on a preset three-dimensional coordinate system;

[0009] Training a three-dimensional pose estimation model according to the at least one training sample for at least one round to obtain a trained three-dimensional pose estimation model;

[0010] wherein, for each training sample, each round of training comprises:

[0011] inputting a sample image of the training sample into a current three-dimensional pose estimation model to obtain a three-dimensional Gaussian mixture representation of each key point on the preset three-dimensional coordinate system, the three-dimensional Gaussian mixture representation being used to represent a probability distribution of a three-dimensional coordinate of a corresponding key point on the preset three-dimensional coordinate system on the sample image;

[0012] for each key point, obtaining a two-dimensional Gaussian mixture representation of the key point on each two-dimensional plane perpendicular to each other in the preset three-dimensional coordinate system according to the three-dimensional Gaussian mixture representation of the key point, the two-dimensional Gaussian mixture representation being used to represent a probability distribution of a two-dimensional coordinate of a corresponding key point on a corresponding two-dimensional plane;

[0013] for each key point, determining a two-dimensional Gaussian distribution of the two-dimensional coordinate of the key point on a corresponding two-dimensional plane according to a two-dimensional actual coordinate of the key point on each two-dimensional plane and a preset standard deviation;

[0014] for each key point, determining a loss value of the key point on each two-dimensional plane according to a difference between the two-dimensional Gaussian mixture representation and the two-dimensional Gaussian distribution of the key point on each two-dimensional plane;

[0015] adjusting a model parameter of the current three-dimensional pose estimation model based on the loss value of each key point on each two-dimensional plane.

[0016] According to a second aspect of the embodiments of the present application, a three-dimensional pose estimation method is provided, comprising:

[0017] obtaining an image;

[0018] determining a region containing a human body in the image by performing human body detection on the image;

[0019] cropping the region containing the human body from the image to obtain a target image;

[0020] determining a three-dimensional coordinate of at least one key point from the target image based on a trained three-dimensional pose estimation model; wherein the three-dimensional pose estimation model is trained based on the model training method of the present application.

[0021] According to a third aspect of the embodiments of the present application, a model training device is provided, comprising:

[0022] a first obtaining module configured to obtain at least one training sample, each training sample comprising a sample image of a human body and annotation information, the annotation information comprising a three-dimensional actual coordinate of at least one key point of the human body on a preset three-dimensional coordinate system in the sample image;

[0023] a first training module configured to perform at least one round of training on the three-dimensional pose estimation model according to the at least one training sample, to obtain a trained three-dimensional pose estimation model;

[0024] wherein, for each training sample, each round of training comprises:

[0025] inputting a sample image of the training sample into the current three-dimensional pose estimation model, to obtain a three-dimensional Gaussian mixture representation of each key point on the preset three-dimensional coordinate system, the three-dimensional Gaussian mixture representation being used to represent a probability distribution of a three-dimensional coordinate of the corresponding key point on the preset three-dimensional coordinate system in the sample image;

[0026] for each key point, obtaining a two-dimensional Gaussian mixture representation of the key point on each two-dimensional plane perpendicular to each other in the preset three-dimensional coordinate system according to the three-dimensional Gaussian mixture representation of the key point, the two-dimensional Gaussian mixture representation being used to represent a probability distribution of a two-dimensional coordinate of the corresponding key point on the corresponding two-dimensional plane;

[0027] for each key point, determining a two-dimensional Gaussian distribution of the two-dimensional coordinate of the key point on the corresponding two-dimensional plane according to a two-dimensional actual coordinate of the key point on each two-dimensional plane and a preset standard deviation;

[0028] for each key point, determining a loss value of the key point on each two-dimensional plane according to a difference between the two-dimensional Gaussian mixture representation and the two-dimensional Gaussian distribution of the key point on each two-dimensional plane;

[0029] adjusting a model parameter of the current three-dimensional pose estimation model based on the loss value of each key point on each two-dimensional plane.

[0030] According to a fourth aspect of the embodiments of the present application, a three-dimensional pose estimation device is provided, comprising:

[0031] a first detection module configured to obtain an image, and determine a region containing a human body in the image by performing human body detection on the image;

[0032] a second obtaining module configured to crop the region containing the human body from the image to obtain a target image;

[0033] a third obtaining module configured to determine a three-dimensional coordinate of at least one key point from the target image based on a trained three-dimensional pose estimation model, wherein the three-dimensional pose estimation model is trained based on the model training method of the present application.

[0034] According to a fifth aspect of the embodiments of the present application, an electronic device is provided, which comprises a memory, a processor and a computer program stored in the memory, and the processor implements the steps of the method provided in the first aspect and the second aspect when executing the program.

[0035] According to a sixth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method provided in the first aspect and the second aspect.

[0036] According to a seventh aspect of the embodiments of the present application, a computer program product is provided, which comprises computer instructions stored in a computer readable storage medium, and when a processor of a computer device reads the computer instructions from the computer readable storage medium, the processor executes the computer instructions to make the computer device execute the steps of the method provided in the first aspect and the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced.

[0038] FIG. 1 is a schematic diagram of a system architecture for implementing model training according to an embodiment of the present application;

[0039] FIG. 2 is a schematic diagram of a model training method according to an embodiment of the present application;

[0040] FIG. 3 is a schematic diagram of one-dimensional Gaussian mixture representation according to an embodiment of the present application;

[0041] FIG. 4 is a schematic diagram of obtaining a sample image according to an embodiment of the present application;

[0042] FIG. 5 is a schematic diagram of a three-dimensional pose estimation method according to an embodiment of the present application;

[0043] FIG. 6 is a schematic diagram of obtaining a three-dimensional coordinate of at least one key point of a target image according to an embodiment of the present application;

[0044] FIG. 7 is a schematic diagram of a three-dimensional pose estimation method applied to palmprint recognition according to an embodiment of the present application;

[0045] FIG. 8 is a schematic diagram of a three-dimensional pose estimation method applied to action recognition according to an embodiment of the present application;

[0046] FIG. 9 is a schematic diagram of a model training device according to an embodiment of the present application;

[0047] FIG. 10 is a schematic diagram of a three-dimensional pose estimation device according to an embodiment of the present application;

[0048] FIG. 11 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0049] Embodiments of the present application will be described below in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions of the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0050] Those skilled in the art can understand that the singular forms "a", "an" and "the" used herein include plural forms, unless specifically stated otherwise. It should be further understood that the terms "comprise" and "include" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the present technology. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or can mean that the element and the other element are connected through an intermediate element. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The term "and / or" used herein means that at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0051] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or a memory) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the functions of the module or unit.

[0052] In order to make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0053] First, several terms related to the present application are introduced and explained:

[0054] Human body detection: refers to using target detection technology to determine the area where the human body is located from the picture, so that the human body area picture can be extracted from the picture.

[0055] Hand detection: refers to the use of object detection techniques to locate the area where the hand is in the picture, so that the hand area picture can be extracted from the picture.

[0056] Human pose estimation: refers to the estimation of key point coordinates of human skeletons in various poses. Human pose estimation usually includes whole body pose estimation and local limb pose estimation. Human pose estimation aims to predict the position information of predefined key points on the human body, is a basic task in computer vision, and is widely used in various visual tasks. It is an important preprocessing operation for many downstream tasks such as human motion analysis, activity recognition, motion capture, etc.

[0057] Hand pose estimation: refers to the estimation of key point coordinates of hand skeletons in various poses; hand pose estimation aims to predict the position information of predefined key points on the hand, is a basic task in computer vision, and is widely used in various visual tasks. It is an important preprocessing operation for many downstream tasks such as gesture recognition, hand motion analysis, motion capture, etc.

[0058] Regression-based pose estimation: for input images, the model directly outputs the coordinates of key points in a regression manner.

[0059] Gaussian mixture representation: a method for describing data distribution, commonly used in statistical modeling and machine learning. In this method, it is assumed that the data is a mixture of multiple Gaussian distributions, each Gaussian distribution represents a sub-group or component of the data. Each Gaussian distribution is described by its mean and covariance, and the mixing proportion represents the proportion of each component in the overall data distribution.

[0060] Linear layer: a neural network layer that performs linear transformation on input.

[0061] Probability distribution: a distribution with a sum of 1, and the value of each point represents the probability corresponding to the point.

[0062] Gaussian mixture model: a probability distribution model composed of a linear combination of multiple Gaussian distribution functions.

[0063] Monte Carlo estimation: a method of approximating numerical calculation by random sampling from a probability model.

[0064] Pearson correlation coefficient: used to measure the degree of correlation between two variables, with a value between -1 and 1.

[0065] Heatmap-based pose estimation: for input images, the model outputs the corresponding heat map to generate the coordinates of key points.

[0066] argmax function: obtains the array index corresponding to the maximum value element in the input array.

[0067] Human pose estimation is an important research direction in the field of computer vision, and is also an indispensable step for computer to understand human action and behavior. Human pose estimation refers to locating human key points in an image or a video through a computer algorithm. At present, human pose estimation is widely applied to many fields such as action detection, virtual reality, human-computer interaction, video tracking and the like.

[0068] In the related art, a pose estimation model can be used to perform pose estimation on a picture to be recognized, to obtain coordinate information of each key point for describing the pose of an object, and then the pose of the related object can be analyzed by using the obtained coordinate information.

[0069] In the process of selecting a scheme for implementing pose estimation, it is assumed that a heat map-based pose estimation scheme is used for processing. In the heat map-based pose estimation scheme, a prediction head generates a high-resolution likelihood heat map according to an input feature map. The number of heat maps is equal to the number of key points to be predicted. This way of generating a corresponding heat map for each key point will occupy a large amount of memory and generate a large amount of calculation cost. Especially when calculating a 3D heat map, the calculation amount will increase by tens of times compared with a 2D heat map. Such a large amount of calculation is difficult to apply to real-time scenarios with high speed requirements and Internet of Things devices with limited computing resources. Moreover, due to the limited size of the heat map, the key point coordinates obtained by using the argmax function often have quantization errors, which will also affect the final performance of the model.

[0070] In the process of selecting a scheme for implementing pose estimation, it is assumed that a traditional regression-based pose estimation scheme is used for processing. In the traditional regression-based pose estimation scheme, a global average pooling is used to simplify the features, and only a plurality of linear layers are included in the prediction head, and the three-dimensional coordinates of the predicted key points are directly output by using the regression manner. However, in the traditional regression-based pose estimation scheme, the directly-regressed three-dimensional coordinate values (vectors) and the input image are not in the same space dimension, so the constraint on the coordinate values is an implicit and non-aligned constraint manner. The model cannot well capture the inherent information in the image, and the obtained performance is poor.

[0071] The model training method and device, electronic equipment, computer readable storage medium and computer program product provided by the present application aim to solve the above technical problems of the related art.

[0072] The technical solutions of the embodiments of the present application and the technical effects generated by the technical solutions of the present application will be described below through the description of several exemplary embodiments. It should be pointed out that the following embodiments can be mutually referenced, borrowed or combined. For the same terms, similar features and similar implementation steps in different embodiments, they will not be described repeatedly.

[0073] FIG. 1 is a schematic diagram of a system architecture for implementing model training according to an embodiment of the present application, which includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected through a wired or wireless network. Optionally, the terminal 101 can be a personal computer (PC), a vehicle-mounted terminal, a tablet computer, a smart phone, a wearable device, a smart robot, or any terminal with data computing, processing and storage capabilities. The server 102 can be a standalone physical server, a server cluster composed of multiple physical servers, a distributed system, or a cloud server providing cloud computing services.

[0074] In the embodiments of the present application, the terminal 101 can be configured to acquire a plurality of to-be-recognized images, each of which includes a human body imaging region, and send the plurality of to-be-recognized images to the server 102. It should be noted that the plurality of to-be-recognized images acquired by the terminal 101 contain depth-of-field information.

[0075] In the embodiments of the present application, the server 102 can perform human body detection on a plurality of to-be-recognized images, determine a region containing a human body in the to-be-recognized images and three-dimensional actual coordinates of each key point of the human body on a preset three-dimensional coordinate system. In the embodiments of the present application, the server 102 can construct the three-dimensional coordinates of each key point on the preset three-dimensional coordinate system according to the two-dimensional coordinates of each key point on the image and the depth information. For each to-be-recognized image, the server 102 crops the region containing the human body from the to-be-recognized image as a sample image of a training sample, and takes the three-dimensional actual coordinates of each key point of the human body on the preset three-dimensional coordinate system as the labeling information of the corresponding training sample. The server 102 inputs each training sample into a three-dimensional pose estimation model for training to obtain a trained three-dimensional pose estimation model. Each round of training includes: inputting the sample image of the training sample into the current three-dimensional pose estimation model to obtain a three-dimensional Gaussian mixture representation of each key point on the preset three-dimensional coordinate system, the three-dimensional Gaussian mixture representation being used to represent the probability distribution of the three-dimensional coordinates of the corresponding key point on the preset three-dimensional coordinate system on the sample image; for each key point, obtaining a two-dimensional Gaussian mixture representation of the key point on three mutually perpendicular two-dimensional planes in the preset three-dimensional coordinate system according to the three-dimensional Gaussian mixture representation of the key point, the two-dimensional Gaussian mixture representation being used to represent the probability distribution of the two-dimensional coordinates of the corresponding key point on the corresponding two-dimensional plane; for each key point, determining a two-dimensional Gaussian distribution of the two-dimensional coordinates of the key point on the corresponding two-dimensional plane according to the two-dimensional actual coordinates of the key point on each two-dimensional plane and a preset standard deviation; for each key point, determining a loss value of the key point on each two-dimensional plane according to the difference between the two-dimensional Gaussian mixture representation and the two-dimensional Gaussian distribution of the key point on each two-dimensional plane; and adjusting the model parameters of the current three-dimensional pose estimation model based on the loss values of the key points on each two-dimensional plane.

[0076] The terminal 101 can also be configured to acquire an image containing a human object, and send the image to the server 102. The server 102 performs human body detection on the image to determine a region containing a human body in the image. The server 102 crops the region containing the human body from the image to obtain a target image, and determines the three-dimensional actual coordinates of at least one key point from the target image based on the trained three-dimensional pose estimation model.

[0077] Those skilled in the art can know that the number of the terminals described above can be more or less. For example, the terminals described above can be only one, or the terminals described above can be tens or hundreds, or more. The number of terminals and the type of equipment are not limited in the embodiments of the present application.

[0078] In some embodiments, the server 102 is a stand-alone physical server, can also be a server cluster or a distributed system composed of multiple physical servers, and can also be a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content delivery network (CDN), and big data and artificial intelligence platform. The server 102 is used to provide background services for an application program supporting image acquisition. In some embodiments, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or the server 102 and the terminal 101 adopt a distributed computing architecture for collaborative computing.

[0079] The following describes scenarios involving pose estimation in combination with several possible application scenarios:

[0080] Scenario one: positioning a region to be identified in an identity recognition process.

[0081] In the application scenario corresponding to scenario one, the recognition information on which the identity recognition is based is determined first, and then the key points to be estimated in the pose estimation are determined according to the required recognition information.

[0082] For example, if palm print is used for identity recognition, the determined key points can at least position the hand region. For another example, if iris is used for identity recognition, the determined key points can at least position the eye region. For another example, if hand gesture is used for identity recognition, the determined key points can at least determine different hand gestures.

[0083] After the server 102 trains the three-dimensional pose estimation model, the three-dimensional pose estimation model can output the coordinate information of the key points based on the image to be identified; then, the region to be identified required for identity recognition can be determined based on the coordinate information of the key points, and the region to be identified can be cropped from the image to be identified.

[0084] Scenario two: action recognition in an anomaly detection process.

[0085] In the application scenario corresponding to scenario two, the object to which the action recognition is directed is determined first, wherein the object to which the action recognition is directed can be a living person or animal, or can be a non-living product that presents different actions with mechanical movement. Then, the key points for three-dimensional pose positioning are determined for the object to which the action recognition is directed, and the three-dimensional pose estimation model is trained using the training samples.

[0086] After obtaining the trained three-dimensional pose estimation model, a target detection technology is first used to detect a region containing a human body from multiple images, and a region containing a human body is then cropped from a to-be-identified image as a target image. Then, the three-dimensional pose estimation model is used to perform pose estimation on the target image to determine respective predicted coordinates of each key point. Then, according to the predicted coordinates, abnormal action recognition (such as falling down) and the like is implemented.

[0087] The model training method and the three-dimensional pose estimation method in the embodiments of the present application can be completed by a server or the like. The scheme in the embodiments of the present application is described below with the server as the execution subject.

[0088] FIG. 2 is a flowchart of a model training method provided by the embodiments of the present application. As shown in FIG. 2, the method includes the following steps.

[0089] S201, at least one training sample is obtained, and each training sample includes a sample image of a human body and annotation information, wherein the annotation information includes three-dimensional actual coordinates of at least one key point of the human body in a preset three-dimensional coordinate system.

[0090] In the embodiments of the present application, before training the three-dimensional pose estimation model, at least one training sample is needed, and each training sample includes a sample image and annotation information. The three-dimensional pose estimation model is iteratively trained through the sample image and the annotation information.

[0091] In the embodiments of the present application, the preset three-dimensional coordinate system can be a three-dimensional Cartesian coordinate system, also known as a three-dimensional world coordinate system, that is, the coordinate system includes an X axis, a Y axis and a Z axis. Correspondingly, any two coordinate axes can form a two-dimensional plane, that is, an XOY plane, an XOZ plane and a YOZ plane. In the embodiments of the present application, for each key point, the coordinates of the key point on the X axis and the Y axis in the sample image can be obtained, and the depth of field is also considered in the embodiments of the present application, so that the coordinates of the key point on the Z axis can be determined according to the depth of field of the key point.

[0092] In the embodiments of the present application, the three-dimensional actual coordinates of the key point can be determined according to the collected image with depth information (i.e., depth data).

[0093] In the embodiments of the present application, the pixel point at the upper left corner of the sample image can be taken as the origin of the preset three-dimensional coordinate system, the X coordinate axis is positive to the right, the Y coordinate axis is positive downward, and the Z coordinate axis is positive upward; the pixel coordinates of each pixel point in the sample image are the positions of the pixel point relative to the origin, which are usually represented by (x, y), where x represents the number of pixels in the horizontal direction, and y represents the number of pixels in the vertical direction. The actual coordinates of each pixel point are the coordinates relative to the physical size of the image or other reference standards. The actual coordinates of each pixel point are the actual coordinates of the key point on the XOY plane in the embodiments of the present application. In the embodiments of the present application, the pixel coordinates of the key point can be converted into the actual coordinates on the XOY plane according to the resolution (the number of pixels per inch) of the sample image and the actual size (for example, the width and the height) of the image. After obtaining the actual coordinates of the key point on the XOY plane, the actual coordinates of the key point on the Z axis can be determined in combination with the depth information of the key point, so as to obtain the three-dimensional actual coordinates of the key point.

[0094] It should be noted that in the embodiments of the present application, the key points can be selected according to the needs of pose estimation. In feasible embodiments, in the case of pose estimation for a “person”, the key points can include key points (or human skeleton key points) of the human body. For example, the key points include joints and facial features. Alternatively, the key points include both the key points of the human body and other self-defined key points; in the case of pose estimation for a part of the human body, the key points can include key points of the local region of the human body, such as in the case of gesture recognition, the key points include general hand key points (including finger joint key points, etc.). In the embodiments of the present application, how to determine which key points are the key points of the whole or local region of the human body is not limited.

[0095] S202, at least one round of training is performed on the three-dimensional pose estimation model according to the at least one training sample, to obtain a trained three-dimensional pose estimation model. In the embodiments of the present application, the three-dimensional pose estimation model includes, for example, a feature extraction network and a regression model. The feature extraction network is used to extract features of an image. The regression model is used to generate three-dimensional predicted coordinates of key points and three-dimensional covariance matrices of a plurality of three-dimensional Gaussian distribution components and weights of each three-dimensional Gaussian distribution component according to the features of the image. For example, the feature extraction network includes a convolutional neural network, such as a residual neural network (ResNet) or the like. In some embodiments, the regression model is a Gaussian mixture regression model. The Gaussian mixture regression model includes a Gaussian mixture model. The Gaussian mixture model is a probability distribution model composed of a linear combination of a plurality of Gaussian distribution functions.

[0096] For each training sample, each round of training includes:

[0097] S2021, input a sample image of the training sample into the current three-dimensional pose estimation model to obtain a three-dimensional Gaussian mixture representation of each key point on the preset three-dimensional coordinate system, the three-dimensional Gaussian mixture representation being used to represent a probability distribution of a three-dimensional coordinate of a corresponding key point on the sample image on the preset three-dimensional coordinate system.

[0098] In the embodiment of the present application, the sample image is input into the three-dimensional pose estimation model of the current iteration (i.e., the current three-dimensional pose estimation model), and the three-dimensional predicted coordinates and the three-dimensional Gaussian mixture representation of each key point can be obtained. The three-dimensional Gaussian mixture representation is used to represent the probability distribution of the three-dimensional coordinate of the corresponding key point on the sample image on the preset three-dimensional coordinate system. It should be noted that in the embodiment of the present application, for each key point, the three-dimensional Gaussian mixture representation of the key point includes a plurality of three-dimensional Gaussian distribution components, each three-dimensional Gaussian distribution component can be regarded as a three-dimensional Gaussian distribution, and different three-dimensional Gaussian distributions use the three-dimensional predicted coordinates of the key point as the mean value, but different three-dimensional Gaussian distributions use different standard deviations. In the embodiment of the present application, the three-dimensional Gaussian mixture representation is obtained by mixing the plurality of three-dimensional Gaussian distribution components according to a preset proportion.

[0099] It should be noted that in the embodiment of the present application, each three-dimensional Gaussian distribution component can be determined by the covariance matrix of the three-dimensional Gaussian distribution component and the weight of the three-dimensional Gaussian distribution component in all three-dimensional Gaussian distribution components.

[0100] In the specific implementation process, given a sample image, it is assumed that the three-dimensional coordinates of K key points need to be predicted When the sample image is input into the current three-dimensional pose estimation model, the three-dimensional covariance matrix Σ 3d,i and the weight w i of the i-th three-dimensional Gaussian distribution component of each key point can be obtained, where 1≤i≤N, N is the number of three-dimensional Gaussian distribution components of each key point, and the value range of the weight w i is (-1, 1).

[0101] It can be understood that the three-dimensional Gaussian mixture representation of each key point is obtained by mixing N three-dimensional Gaussian distribution components according to their respective weights. The three-dimensional covariance matrix Σ 3d,i of the i-th Gaussian distribution component can be decomposed as follows:

[0102] wherein are the standard deviations of the i-th Gaussian distribution component on the X-axis, the Y-axis and the Z-axis, respectively, and the standard deviation of each three-dimensional Gaussian distribution component on each coordinate axis is used to represent the average deviation of the three-dimensional coordinate of each key point on the sample image from the three-dimensional predicted coordinate on the corresponding coordinate axis. The Pearson correlation coefficient Pearson correlation coefficient for representing the correlation between the standard deviation of the X-axis and the Y-axis Pearson correlation coefficient for representing the correlation between the standard deviation of the X-axis and the Y-axis Pearson correlation coefficient for representing the correlation between the standard deviation of the X-axis and the Z-axis Pearson correlation coefficient for representing the correlation between the standard deviation of the Y-axis and the Z-axis Pearson correlation coefficient for representing the correlation between the standard deviation of the Y-axis and the Z-axis The value range of each of the Pearson correlation coefficients is (-1, 1).

[0103] In the embodiments of the present application, by means of three-dimensional Gaussian mixture representation, the three-dimensional pose estimation model converts the coordinates of the key points into a probability distribution of three-dimensional coordinates in a preset three-dimensional coordinate system, so that the output three-dimensional coordinates of the key points and the input sample cloth pose image are in the same spatial dimension. In this way, the representation ability of the three-dimensional pose estimation model in the sample image is improved, and the model obtains better performance.

[0104] S2022, for each key point, obtaining a two-dimensional Gaussian mixture representation of the key point on three mutually perpendicular two-dimensional planes in the preset three-dimensional coordinate system according to the three-dimensional Gaussian mixture representation of the key point, the two-dimensional Gaussian mixture representation being used to represent a probability distribution of two-dimensional coordinates of the corresponding key point on the corresponding two-dimensional plane.

[0105] In the embodiments of the present application, after obtaining the three-dimensional Gaussian mixture representation of each key point, the three-dimensional Gaussian mixture representation can be decomposed to obtain the two-dimensional Gaussian mixture representation of each key point on the XOY plane, the XOZ plane and the YOZ plane, and the two-dimensional Gaussian mixture representation of each key point on any two-dimensional plane is used to represent a probability distribution of two-dimensional coordinates of the corresponding key point on the corresponding two-dimensional plane.

[0106] In the specific implementation process, the three-dimensional covariance matrix of each key point can be decomposed into two-dimensional covariance matrices on three two-dimensional planes. The two-dimensional covariance matrix on each two-dimensional plane is as follows:

[0107] wherein, Σ 12 represents the covariance matrix of the key point on the XOY plane; Σ 13 represents the covariance matrix of the key point on the XOZ plane; and Σ 23 represents the covariance matrix of the key point on the YOZ plane.

[0108] It should be noted that the above two-dimensional covariance matrix is determined according to the i-th three-dimensional Gaussian distribution component of the key point. For each key point, the determination method of the two-dimensional covariance matrix of other three-dimensional Gaussian distribution components is the same, and the embodiments of the present application will not be described again.

[0109] In the embodiments of the present application, for each key point, the two-dimensional predicted coordinates of the key point on the corresponding two-dimensional plane can be obtained through the three-dimensional predicted coordinates of the key point, and the two-dimensional predicted coordinates are the mean values of the two-dimensional Gaussian distribution components of the key point on the corresponding two-dimensional plane, and it can be understood that the plurality of two-dimensional Gaussian distribution components of the key point on each two-dimensional plane use the same mean value. In addition, the weight of each two-dimensional Gaussian distribution component is the same as the weight of the corresponding three-dimensional Gaussian distribution component, and it can be understood that the weight w i of the i-th three-dimensional Gaussian distribution component of each key point is w i .

[0110] In the embodiments of the present application, for each key point, after obtaining the two-dimensional predicted coordinates, the two-dimensional covariance matrix and the weight of each two-dimensional Gaussian distribution component of the key point on each two-dimensional plane, the plurality of two-dimensional Gaussian distribution components of the key point on each two-dimensional plane can be determined. Wherein, assuming that the two-dimensional predicted coordinates of the key point on the XOY plane are the two-dimensional covariance of the i-th two-dimensional Gaussian distribution component of the key point is Σ i and the weight is w i , then the i-th two-dimensional Gaussian distribution component of the key point can be defined by a parameter group, which can be represented as:

[0111] It should be noted that each two-dimensional Gaussian component is regarded as a probability density function, and the Gaussian mixture representation p θ (x) of the key point on the XOY plane is obtained by weighted summation of the weights of each two-dimensional Gaussian distribution component. Similarly, the two-dimensional Gaussian mixture representations of the key point on the XOZ and YOZ planes can be obtained. θ (x) can be represented by the following formula:

[0112] Wherein, p θ (x) is the two-dimensional Gaussian mixture representation of any key point on the XOY plane; N is the total number of two-dimensional Gaussian distribution components of the key point on the XOY plane; w i represents the weight of the i-th Gaussian distribution component of the key point on the XOY plane; Σ i is the two-dimensional covariance matrix of the i-th Gaussian distribution component of the key point on the XOY plane; represents the mean value determined according to the two-dimensional predicted coordinate of the key point on the XOY plane; x is a variable, representing the coordinate of any one pixel point in the corresponding sample image on the XOY plane.

[0113] In the embodiments of the present application, by decomposing the three-dimensional Gaussian mixture representation of the key point into two-dimensional Gaussian mixture representations on three two-dimensional planes, the direct construction of the three-dimensional Gaussian mixture representation to constrain the coordinates of the key point is avoided, the problem that the non-invertibility of the covariance matrix causes the gradient back propagation to be wrong, and further causes the model to be unable to correctly learn and optimize the parameters is solved, and the performance and accuracy of the three-dimensional pose estimation model are improved.

[0114] S2023, for each key point, determining a two-dimensional Gaussian distribution of the two-dimensional coordinate of the key point on the corresponding two-dimensional plane according to the two-dimensional actual coordinate of the key point on each two-dimensional plane and a preset standard deviation.

[0115] S2024, for each key point, determining a loss value of the key point on each two-dimensional plane according to the difference between the two-dimensional Gaussian mixture representation and the two-dimensional Gaussian distribution of the key point on each two-dimensional plane.

[0116] In the embodiments of the present application, for each key point, the two-dimensional actual coordinates of the key point on three coordinate axes can be determined according to the three-dimensional actual coordinates of the key point. In addition, in the embodiments of the present application, for each key point, the Gaussian distribution of the key point on the corresponding two-dimensional plane can be determined according to the two-dimensional actual coordinates of the key point on each two-dimensional plane and the preset standard deviation on the corresponding two-dimensional plane, denoted as p g (x), where x is a variable, representing the coordinate of any one pixel point in the corresponding sample image on the corresponding two-dimensional plane.

[0117] In the embodiments of the present application, the loss value of the key point on each two-dimensional plane is determined according to the distribution difference between the two-dimensional Gaussian mixture representation and the two-dimensional Gaussian distribution of the key point on each two-dimensional plane Loss value which can be represented by the following formula:

[0118] where p g is the Gaussian distribution of the key point on the two-dimensional plane; p θ is the two-dimensional Gaussian mixture representation; D KL represents the KL divergence between the two-dimensional Gaussian mixture representation of the key point on each two-dimensional plane and the two-dimensional Gaussian distribution on the corresponding two-dimensional plane, denoted as the first loss; ||p g ,p θ||1 represents the L1 loss between the two-dimensional Gaussian mixture representation of the key point on each two-dimensional plane and the two-dimensional Gaussian distribution on the corresponding two-dimensional plane, denoted as the second loss; and γ is a smoothing coefficient of the smooth L1 loss, which is set according to actual processing needs.

[0119] In order to facilitate those skilled in the art to intuitively understand the method of constructing the Gaussian mixture representation of the key point, the following will be described by taking the construction of the one-dimensional Gaussian mixture representation of the key point as an example.

[0120] FIG. 3 is a schematic diagram of the one-dimensional Gaussian mixture representation provided by the embodiment of the present application. As shown in FIG. 3, after the sample image is input into the three-dimensional pose estimation model, the three-dimensional pose estimation model outputs the predicted coordinate of the key point on the X axis the covariance matrix Σ of the L one-dimensional Gaussian distribution components i the weight w of each one-dimensional Gaussian distribution component i ; wherein i represents any one-dimensional Gaussian distribution component, and the value range of i is 1≤i≤N. For any one-dimensional Gaussian distribution component, according to the predicted coordinate the covariance matrix of the one-dimensional Gaussian distribution component, and the weight of the one-dimensional Gaussian distribution component, the one-dimensional Gaussian distribution component is determined; and all one-dimensional Gaussian distribution components are subjected to Gaussian mixture processing to obtain the one-dimensional Gaussian mixture representation p θ of the key point on the X axis. At the same time, according to the actual coordinate of the key point on the X axis of the sample image and the preset standard deviation, the one-dimensional Gaussian distribution p g of the key point on the X axis can be determined. By comparing the difference between the one-dimensional Gaussian mixture representation p θ and the one-dimensional Gaussian distribution p g , the loss value of the key point on the X axis is determined. According to the same principle, the Gaussian mixture representation of the key point on any two-dimensional plane can be obtained.

[0121] S2025, based on the loss value of each key point on each two-dimensional plane, the model parameter of the current three-dimensional pose estimation model is adjusted.

[0122] In the embodiment of the present application, the server 102 can adjust the model parameter of the current three-dimensional pose estimation model according to the loss value of each key point on each two-dimensional plane.

[0123] It should be noted that in the embodiment of the present application, if the loss value of each key point on each two-dimensional plane satisfies the preset convergence condition, the iterative training of the three-dimensional pose estimation model is stopped.

[0124] In some embodiments, the preset convergence condition can be that the total number of training rounds reaches a first threshold, or the number of times that the calculated loss value of each key point on each two-dimensional plane is continuously lower than a second threshold reaches a third threshold, wherein the values of the first threshold, the second threshold, and the third threshold are set according to actual processing needs.

[0125] The model training method provided in the embodiments of the present application converts the constraint on the coordinates of the key points by directly constructing a three-dimensional Gaussian mixture representation in the related art into the decomposition of the three-dimensional Gaussian mixture representation into two-dimensional Gaussian mixture standards on three two-dimensional planes, solves the problem that the covariance matrix of the three-dimensional Gaussian mixture representation in the related art is not reversible, causing the gradient back propagation to be incorrect, and effectively improves the accuracy of three-dimensional pose estimation. Meanwhile, the multiple three-dimensional Gaussian distributions used in the model training stage are used as plug-ins connected to the model; therefore, in the model application stage, the plug-ins can be directly removed and do not participate in the overall time consumption, which is suitable for real-time human pose estimation scenarios.

[0126] In addition, in the embodiments of the present application, the parameters of the three-dimensional Gaussian mixture representation of the key points are directly predicted by the three-dimensional pose estimation model, each three-dimensional Gaussian mixture representation is mixed according to a certain proportion from multiple three-dimensional Gaussian distributions, and the multiple three-dimensional Gaussian distributions can be used as plug-ins connected to the three-dimensional pose estimation model; therefore, in the model application stage, the plug-ins can be directly removed and do not participate in the overall time consumption. Compared with the related art, the three-dimensional pose estimation model of the present application is not only lighter, but also does not bring additional time consumption burden.

[0127] On the basis of the above embodiments, as an optional embodiment, the loss value of the key point on each two-dimensional plane is determined according to the difference between the two-dimensional Gaussian mixture representation of the key point on each two-dimensional plane and the two-dimensional Gaussian distribution, including:

[0128] For each key point, the KL divergence loss between the two-dimensional Gaussian mixture representation of the key point on each two-dimensional plane and the corresponding two-dimensional Gaussian distribution is taken as the first loss of the key point on the corresponding two-dimensional plane.

[0129] For each key point, the L1 loss between the two-dimensional Gaussian mixture representation of the key point on each two-dimensional plane and the corresponding two-dimensional Gaussian distribution is taken as the second loss of the key point on the corresponding two-dimensional plane.

[0130] For each key point, the loss value of the key point on the corresponding two-dimensional plane is determined according to the first loss and the second loss of the key point on each two-dimensional plane.

[0131] The embodiments of the present application relate to using a two-dimensional Gaussian mixture model to represent the distribution of a key point on each two-dimensional plane, and using KL divergence loss and L1 loss to measure the difference between the two-dimensional Gaussian mixture representation and the two-dimensional Gaussian distribution. Among them, the two-dimensional Gaussian distribution is a normal distribution, the mean of the two-dimensional Gaussian distribution is the two-dimensional actual coordinate of the key point on the corresponding plane, and the standard deviation of the two-dimensional Gaussian distribution on each coordinate axis uses a preset value.

[0132] In the embodiments of the present application, for each key point, the KL divergence loss between the two-dimensional Gaussian mixture model of the key point on each two-dimensional plane and the two-dimensional Gaussian distribution can be used to evaluate the fitting degree between the two-dimensional Gaussian mixture representation and the two-dimensional Gaussian distribution. Similarly, for each key point, the L1 loss between the two-dimensional Gaussian mixture model of the key point on each two-dimensional plane and the corresponding two-dimensional Gaussian distribution can be obtained. The average value of the absolute difference between each point of the two-dimensional Gaussian mixture model and the corresponding two-dimensional Gaussian distribution.

[0133] In the embodiments of the present application, according to the calculated KL divergence loss and L1 loss, the loss value of each key point on the corresponding two-dimensional plane is determined.

[0134] In the embodiments of the present application, the KL divergence loss and the L1 loss can be weighted or combined according to the actual situation to obtain the final loss value.

[0135] It should be noted that in the embodiments of the present application, due to the instability of the value of the KL divergence, especially the certain fluctuation on the zero probability density of the distribution; therefore, when determining the loss value, an additional L1 loss is added based on the calculation of the KL divergence.

[0136] The embodiments of the present application utilize the probability distribution comparison method in statistics to evaluate the difference between the two-dimensional Gaussian mixture representation and the two-dimensional Gaussian distribution of each key point on any two-dimensional plane, and optimize these differences as part of the loss function.

[0137] On the basis of the above embodiments, as an optional embodiment, for each key point, according to the first loss and the second loss of the key point on each two-dimensional plane, the loss value of the key point on the corresponding two-dimensional plane is determined, comprising:

[0138] For each key point, the second loss is smoothed according to a preset smoothing coefficient to obtain a third loss;

[0139] For each key point, the sum of the third loss and the first loss is taken as the loss value of the key point on each two-dimensional plane.

[0140] In the embodiments of the present application, the loss value of each key point on the corresponding two-dimensional plane is determined according to the first loss and the second loss; the second loss is smoothed using a preset smoothing coefficient to obtain a third loss; wherein the smoothing can use different methods such as weighted average or filter; and the sum of the third loss and the first loss is taken as the loss value of the key point on each two-dimensional plane.

[0141] In the embodiments of the present application, the smoothing of the second loss using the preset smoothing coefficient is part of the loss function design and optimization, which needs to be adjusted and implemented according to the specific problem and the structure of the three-dimensional pose estimation model; at the same time, the selection of smoothing and loss combination will also affect the final optimization effect and model performance.

[0142] On the basis of the above embodiments, as an optional embodiment, the sample image of the training sample is input into the current three-dimensional pose estimation model to obtain a three-dimensional Gaussian mixture representation of each key point on a preset three-dimensional coordinate system, including:

[0143] The sample image is input into the current three-dimensional pose estimation model to obtain a three-dimensional predicted coordinate of each key point on the sample image generated by the three-dimensional pose estimation model, and a three-dimensional covariance matrix of a plurality of three-dimensional Gaussian distribution components and a weight of each three-dimensional Gaussian distribution component in the three-dimensional Gaussian mixture representation;

[0144] For each key point, the three-dimensional Gaussian mixture representation of the key point is determined according to the three-dimensional predicted coordinate of the key point, the three-dimensional covariance matrix of the plurality of three-dimensional Gaussian distribution components and the weight of each three-dimensional Gaussian distribution component.

[0145] Wherein, each three-dimensional Gaussian distribution component is a three-dimensional Gaussian distribution in the three-dimensional Gaussian mixture representation, and the weight of each three-dimensional Gaussian distribution component is the mixing proportion of the three-dimensional Gaussian distribution component in the three-dimensional Gaussian mixture representation.

[0146] In the embodiments of the present application, each three-dimensional Gaussian distribution component represents a three-dimensional Gaussian distribution of the key point on the preset three-dimensional coordinate system, and the parameters of the three-dimensional Gaussian distribution component include the mean value (determined by the three-dimensional predicted coordinate of the key point), the covariance matrix and the weight. The weight of each three-dimensional Gaussian distribution component represents the mixing proportion of the component in the three-dimensional Gaussian mixture representation, which is used to determine the final three-dimensional Gaussian mixture distribution.

[0147] It should be noted that in the embodiments of the present application, for each key point, the weights of the plurality of three-dimensional Gaussian distribution components of the key point can be the same or different, and the embodiments of the present application do not limit the specific value of the weight of each three-dimensional Gaussian distribution component.

[0148] In the embodiments of the present application, for each key point, the three-dimensional Gaussian mixed representation of the key point is determined according to the three-dimensional predicted coordinates of the key point, the three-dimensional covariance matrix of the plurality of three-dimensional Gaussian distribution components, and the weight of each three-dimensional Gaussian distribution component, so as to convert the pose information in the training sample into the three-dimensional Gaussian mixed representation, and provide more accurate model and more abundant information for subsequent three-dimensional pose estimation.

[0149] On the basis of the above embodiments, as an optional embodiment, for each key point, the three-dimensional Gaussian mixed representation of the key point is determined according to the three-dimensional predicted coordinates of the key point, the three-dimensional covariance matrix of the plurality of three-dimensional Gaussian distribution components, and the weight of each three-dimensional Gaussian distribution component, including:

[0150] For each key point, the plurality of three-dimensional Gaussian distribution components of the key point are determined according to the three-dimensional predicted coordinates of the key point and the three-dimensional covariance matrix of the plurality of three-dimensional Gaussian distribution components.

[0151] For each key point, the three-dimensional Gaussian mixed representation of the key point is obtained according to the plurality of three-dimensional Gaussian distribution components of the key point and the weight of each three-dimensional Gaussian distribution component.

[0152] In the embodiments of the present application, for each key point, the plurality of three-dimensional Gaussian distribution components of the key point are determined by using the three-dimensional predicted coordinates of the key point and the three-dimensional covariance matrix of the plurality of three-dimensional Gaussian distribution components. These components represent the uncertainty and variation range of the key point in different directions. At the same time, for each key point, the three-dimensional Gaussian mixed representation of the key point is calculated according to the plurality of three-dimensional Gaussian distribution components of the key point and the weight of each component.

[0153] In the embodiments of the present application, for each key point, the three-dimensional Gaussian mixed representation of the key point is obtained by weighting and summing the plurality of three-dimensional Gaussian distribution components according to their weights, which can more accurately represent the three-dimensional coordinates of each key point and take into account the three-dimensional coordinate uncertainty and variation range of each key point, thereby improving the modeling accuracy and robustness of the three-dimensional pose of each key point.

[0154] On the basis of the above embodiments, as an optional embodiment, the three-dimensional covariance matrix of the plurality of three-dimensional Gaussian distribution components of each key point is determined by the following method, including:

[0155] For each key point, the standard deviation of each three-dimensional Gaussian distribution component of the key point generated by the current three-dimensional pose estimation model on the three coordinate axes of the preset three-dimensional coordinate system and the Pearson correlation coefficient between the standard deviations of two coordinate axes are obtained.

[0156] For each key point, a covariance matrix of each three-dimensional Gaussian distribution component of the key point is determined according to the standard deviation of each three-dimensional Gaussian distribution component of the key point on three coordinate axes and the Pearson correlation coefficient between the standard deviations of two coordinate axes.

[0157] wherein the standard deviation of each three-dimensional Gaussian distribution component on each coordinate axis is used to represent the average deviation degree of the three-dimensional coordinates of each key point on the sample image from the three-dimensional predicted coordinates of the key point on the corresponding coordinate axis.

[0158] In the embodiments of the present application, the standard deviation of each key point on three coordinate axes of a preset three-dimensional coordinate system and the Pearson correlation coefficient between the standard deviations of two coordinate axes reflect the uncertainty and correlation of each key point on different coordinate axes. In addition, the covariance matrix of the three-dimensional Gaussian distribution component can be used to describe the correlation degree between different dimensions, which can reflect the shape and direction of the key point in the three-dimensional space. Finally, for each key point, the standard deviation of each three-dimensional Gaussian distribution component of the key point on each coordinate axis is used to quantify the average deviation degree of the three-dimensional coordinates of each key point on the sample image from the three-dimensional predicted coordinates of the key point on the corresponding coordinate axis, which can more accurately capture the variation range and uncertainty of the key point and provide a more reliable basis for subsequent pose estimation.

[0159] On the basis of the above embodiments, as an optional embodiment, for each key point, a two-dimensional Gaussian mixture representation of the key point on three mutually perpendicular two-dimensional planes in the preset three-dimensional coordinate system is obtained according to the three-dimensional Gaussian mixture representation of the key point, including:

[0160] For each key point, a two-dimensional predicted coordinate of the three-dimensional predicted coordinate on each two-dimensional plane is obtained respectively according to the three-dimensional predicted coordinate of the key point;

[0161] For each key point, a three-dimensional covariance matrix of a plurality of three-dimensional Gaussian distribution components of the key point is obtained, and for each three-dimensional Gaussian distribution component of the key point, a two-dimensional covariance matrix of the three-dimensional covariance matrix on each two-dimensional plane is obtained according to the three-dimensional covariance matrix of the three-dimensional Gaussian distribution component;

[0162] For each key point, a plurality of two-dimensional Gaussian distribution components of the key point on the corresponding two-dimensional plane are determined according to the two-dimensional predicted coordinates of the key point on each two-dimensional plane and the plurality of two-dimensional covariance matrices on the corresponding two-dimensional plane;

[0163] For each key point, a two-dimensional Gaussian mixture representation of the key point on each two-dimensional plane is determined according to the plurality of two-dimensional Gaussian distribution components of the key point on each two-dimensional plane and the weight of each two-dimensional Gaussian distribution component, the weight of each two-dimensional Gaussian distribution component being the same as the weight of the corresponding three-dimensional Gaussian distribution component.

[0164] In the embodiments of the present application, for each key point, a two-dimensional covariance matrix of the three-dimensional covariance matrix of the key point on each two-dimensional plane is calculated; in this way, the uncertainty in the three-dimensional space can be converted into the uncertainty on the two-dimensional plane. At the same time, for each key point, the weight of the plurality of two-dimensional Gaussian distribution components of the key point on each two-dimensional plane is the same as the weight of the corresponding three-dimensional Gaussian distribution component, so as to maintain consistency and accuracy.

[0165] In the embodiments of the present application, for each key point, the three-dimensional Gaussian mixture representation of the key point is decomposed into a two-dimensional Gaussian mixture representation on three mutually perpendicular two-dimensional planes, which overcomes the problem that the covariance matrix of the three-dimensional Gaussian mixture representation is not reversible, causing the gradient back propagation to be wrong, when the related art directly constructs the three-dimensional Gaussian mixture representation to constrain the coordinates of the key point, and effectively improves the accuracy of the three-dimensional pose estimation.

[0166] On the basis of the above embodiments, as an optional embodiment, at least one training sample is obtained, and the above further includes:

[0167] A plurality of images are obtained; each image contains a human body imaging region;

[0168] Human body detection is performed on the plurality of images to determine the region containing the human body in the plurality of images;

[0169] The three-dimensional actual coordinates of each key point of the human body in a preset three-dimensional coordinate system are determined;

[0170] The region containing the human body is cropped from the image to obtain a sample image of a training sample;

[0171] The three-dimensional actual coordinates of each key point of the human body in the preset three-dimensional coordinate system are taken as the labeling information of the corresponding training sample.

[0172] In the embodiments of the present application, the human body object detection is performed on the plurality of images to obtain the region containing the human body in the image, the region containing the human body is cropped from the image as a sample image of a training sample, and the three-dimensional actual coordinates of each key point of the human body in a preset three-dimensional coordinate system are taken as the labeling information of the corresponding training sample, so as to construct a data set containing the training sample and the labeling information, which is used to train the three-dimensional pose estimation model.

[0173] It should be noted that in the examples of the present application, the human object detection and the obtaining of the three-dimensional actual coordinates of each key point of the human body in multiple images belong to the preprocessing step of three-dimensional pose estimation, and the corresponding functions are included in the three-dimensional pose estimation model.

[0174] FIG. 4 is a schematic diagram of obtaining a sample image according to an embodiment of the present application. As shown in FIG. 4, in the case of pose estimation for a “person”, after the processing device obtains an original image, the actual pose estimation needs can be determined, and target detection can be performed on the original image. In the form of a target detection box, the human body region or the local human body region for pose estimation is identified, wherein the detection method used can be a general human body region detection method (such as the YOLO algorithm), or a human body local region detection method. Then, the region marked by the target detection box (denoted as the ROI region) is cropped from the original image as the input of the three-dimensional pose estimation model.

[0175] In this way, by cropping the human body imaging region in the image to be processed, the introduction of background content interference can be avoided as much as possible in the obtained sample image, and the pose estimation effect for the specified object is ensured.

[0176] In addition, the applicant compared the three-dimensional pose estimation method in the related art with the pose estimation method proposed in the present application, and obtained the following comparison results.

[0177] Table 1 Comparison of test results of different pose estimation methods

[0178] Specifically, as shown in Table 1, the effects of the three-dimensional pose estimation model trained by the embodiments of the present application and other methods in the pose estimation field on the validation set of the public data set Human3.6M are given, and the evaluation index is MPJPE and PAMPJPE. The evaluation index is used to represent the accuracy of the model prediction. The lower the evaluation index, the more accurate the model prediction.

[0179] From Table 1, it can be seen that when compared with other methods in the pose estimation field using the same level and size of backbone network, the three-dimensional pose estimation model provided by the embodiments of the present application can exceed the performance of all other methods.

[0180] FIG. 5 is a flowchart of a three-dimensional pose estimation method according to an embodiment of the present application. As shown in FIG. 5, the method comprises the following steps:

[0181] S301, obtaining an image;

[0182] S302, performing human body detection on the image to determine a region containing a human body in the image.

[0183] In the embodiments of the present application, a human body detection algorithm in computer vision technology (such as a human body detection model based on deep learning) can be used to identify the human body region in the image. These algorithms usually return a bounding box or a pixel mask in which the human body is located.

[0184] It should be noted that in the embodiments of the present application, the human body imaging region in the original image includes depth of field information, so the front and back distance range of the whole or part of the human body in the original image can be determined according to the depth of field information. The depth of field information of the human body imaging region in the embodiments of the present application can be obtained by adjusting the aperture size, selecting a suitable lens focal length, adjusting the distance between the camera and the subject, using the depth of field preview function, and post-processing, and the present application does not limit how to obtain the depth of field information of the human body imaging region.

[0185] S303, cropping a region containing a human body from the image to obtain a target image.

[0186] In the embodiments of the present application, after obtaining the position information of the human body imaging region, the original image can be cropped according to the information to obtain a target image containing a human body. For example, the image can be cropped according to the position and size of the bounding box, or using a pixel mask. The cropping result can also be adjusted as needed to ensure that the complete human body is contained and important detail information is not lost.

[0187] S304, determining the three-dimensional coordinates of at least one key point from the target image based on the trained three-dimensional pose estimation model; wherein the three-dimensional pose estimation model is obtained based on the above model training method.

[0188] FIG. 6 is a flowchart of obtaining the three-dimensional coordinates of at least one key point of the target image according to the embodiments of the present application; as shown in FIG. 6, it is assumed that the target image contains a human body imaging region, wherein the human body imaging region is defined with K key points, the target image is input into the trained three-dimensional pose estimation model, and the three-dimensional coordinates (i.e. three-dimensional predicted coordinates) of the K key points are obtained.

[0189] In the embodiments of the present application, after obtaining the three-dimensional coordinates of each key point through the three-dimensional pose estimation model, the three-dimensional pose of the human body in the target image can be determined according to the relationship between the three-dimensional coordinates of each key point.

[0190] In the embodiment of the present application, in the process of applying the three-dimensional pose estimation model to estimate the three-dimensional pose of the human body imaging area, the two-dimensional Gaussian mixture representation of each key point of the human body imaging area on the three two-dimensional planes is not required, and the determination process of the two-dimensional Gaussian mixture representation is only in the training stage of the three-dimensional pose estimation model. After obtaining the trained three-dimensional pose estimation model, the function of outputting the corresponding two-dimensional Gaussian mixture representation of each key point of the model can be regarded as a plug-in connected after the three-dimensional pose estimation model. Therefore, when estimating the three-dimensional pose of the human body imaging area based on the trained three-dimensional pose estimation model, the related plug-in can be directly removed without participating in the overall time consumption, which not only improves the training effect of the three-dimensional pose estimation model, but also reduces the resource occupation burden caused by the application of the trained three-dimensional pose estimation model, thereby ensuring the efficiency of the three-dimensional pose estimation.

[0191] The business processing process involved will be described below with reference to several business processes using the three-dimensional pose estimation model.

[0192] FIG. 7 is a schematic diagram of the three-dimensional pose estimation method provided by the embodiment of the present application applied to palmprint recognition. The process of realizing palmprint recognition based on the three-dimensional pose estimation model will be described below with reference to FIG. 7.

[0193] With the increasing attention of the public to privacy issues, palmprint recognition has a wider application prospect in practical application scenarios such as payment and identity verification. The human pose estimation technology provided by the present application can be applied to the hands of the human body to detect the key points of the hands in real time to complete the positioning of the palm area.

[0194] Specifically, before identifying the palmprint of each user, the palm detection model and the palm key point detection model (i.e., the target pose estimation model) applying the human pose estimation technology can be used to constitute a palm recognition component, and the palm of the user is registered in the background registration library. In each subsequent identification process, after obtaining the captured image, the palm detection and hand pose estimation processing are performed on the captured image in sequence, and finally the hand region is determined in the image, and the hand region is cut from the image. Then, the image of the hand region is compared with each photo in the registration library for palmprint recognition to identify the identity of the user and complete identity verification. After palm detection, the hand region can be roughly determined in the image, and after hand pose estimation processing, the positions of the key points of the hand can be determined, so that the hand region can be accurately positioned.

[0195] FIG. 8 is a schematic diagram of a three-dimensional pose estimation method provided by an embodiment of the present application applied to motion recognition. The three-dimensional pose estimation method provided by the present application can be applied to motion, gesture and gait recognition, such as judging falling conditions and disease signals, automatic teaching of fitness, sports and dance, etc. As shown in the content of FIG. 8, in the process of motion recognition, gesture recognition and gait recognition, the processing logic involved is as follows: after the image is captured, the human body imaging area is detected to locate the human body or hand area; then the three-dimensional pose estimation model is used to realize human body pose estimation to locate the key points of the human body or hand; further, the extraction of the region of interest (ROI) is realized according to the determined key points, and then the subsequent motion, gesture and gait recognition is completed according to the determined region of interest.

[0196] FIG. 9 is a structural schematic diagram of a model training device provided by an embodiment of the present application. As shown in FIG. 9, the model training device can include a first acquisition module 9001 and a first training module 9002.

[0197] The first acquisition module 9001 is configured to acquire at least one training sample, each training sample including a sample image of a human body and annotation information, the annotation information including three-dimensional actual coordinates of at least one key point of the human body in the sample image on a preset three-dimensional coordinate system;

[0198] The first training module 9002 is configured to perform at least one round of training on a three-dimensional pose estimation model according to the at least one training sample to obtain a trained three-dimensional pose estimation model; wherein for each training sample, each round of training includes:

[0199] inputting the sample image of the training sample into the current three-dimensional pose estimation model to obtain a three-dimensional Gaussian mixture representation of each key point on the preset three-dimensional coordinate system, the three-dimensional Gaussian mixture representation being used to represent a probability distribution of three-dimensional coordinates of the corresponding key point on the preset three-dimensional coordinate system on the sample image;

[0200] for each key point, obtaining a two-dimensional Gaussian mixture representation of the key point on each two-dimensional plane perpendicular to each other in the preset three-dimensional coordinate system according to the three-dimensional Gaussian mixture representation of the key point, the two-dimensional Gaussian mixture representation being used to represent a probability distribution of two-dimensional coordinates of the corresponding key point on the corresponding two-dimensional plane;

[0201] for each key point, determining a two-dimensional Gaussian distribution of the two-dimensional coordinates of the key point on the corresponding two-dimensional plane according to the two-dimensional actual coordinates of the key point on each two-dimensional plane and a preset standard deviation;

[0202] For each key point, a loss value of the key point on each two-dimensional plane is determined according to a difference between a two-dimensional Gaussian mixture representation of the key point on each two-dimensional plane and a two-dimensional Gaussian distribution;

[0203] Based on the loss value of each key point on each two-dimensional plane, a model parameter of a current three-dimensional pose estimation model is adjusted.

[0204] The model training apparatus of the embodiments of the present application can perform the model training method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the model training apparatus of the embodiments of the present application correspond to the steps in the model training method of the embodiments of the present application. For detailed descriptions of the functions of each module of the model training apparatus, refer to the descriptions of the corresponding method in the foregoing, which will not be described here.

[0205] FIG. 10 is a structural schematic diagram of a three-dimensional pose estimation apparatus provided by an embodiment of the present application. As shown in FIG. 10, the three-dimensional pose estimation apparatus can include a first detection module 1001, a second acquisition module 1002, and a third acquisition module 1003, wherein

[0206] The first detection module 1001 is configured to acquire an image, and determine a region containing a human body in the image by performing human body detection on the image.

[0207] The second acquisition module is configured to crop the region containing the human body from the image to obtain a target image.

[0208] The third acquisition module is configured to determine a three-dimensional coordinate of at least one key point from the target image based on a trained three-dimensional pose estimation model. The three-dimensional pose estimation model is trained according to the model training method of the embodiments of the present application.

[0209] The three-dimensional pose estimation apparatus of the embodiments of the present application can perform the three-dimensional pose estimation method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the three-dimensional pose estimation apparatus of the embodiments of the present application correspond to the steps in the three-dimensional pose estimation method of the embodiments of the present application. For detailed descriptions of the functions of each module of the three-dimensional pose estimation apparatus, refer to the descriptions of the corresponding method in the foregoing, which will not be described here.

[0210] Fig. 11 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. As shown in Fig. 11, the electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, through a bus 4002. Optionally, the electronic device 4000 can further include a transceiver 4004, which can be used for data interaction, such as data sending and / or data receiving, between the electronic device and other electronic devices. It should be noted that the transceiver 4004 is not limited to one in actual application, and the structure of the electronic device 4000 does not constitute a limitation to the embodiments of the present application.

[0211] The processor 4001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. The processor 4001 can implement or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure of the present application. The processor 4001 can also be a combination for implementing a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0212] The bus 4002 can include a path for transmitting information between the above-mentioned components. The bus 4002 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is shown in Fig. 11, but it does not mean that there is only one bus or only one type of bus.

[0213] The memory 4003 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read-Only Memory), a CD-ROM (Compact Disc Read-Only Memory) or other optical disc storage, an optical disc storage (including a compact disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disc storage medium, other magnetic storage device, or any other medium that can be used to carry or store computer programs and that can be accessed by a computer, without limitation.

[0214] The memory 4003 is configured to store a computer program for implementing the embodiments of the present application, and the processor 4001 is configured to control the execution of the computer program stored in the memory 4003. The processor 4001 is configured to execute the computer program stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0215] The electronic device package can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet PCs), PMPs (Portable Multimedia Players), vehicle terminals (e.g., car navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. The electronic device shown in FIG. 11 is merely an example, and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0216] The embodiments of the present application provide a computer readable storage medium, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the steps and corresponding contents of the foregoing method embodiments can be implemented. Compared with the related art, the following can be achieved:

[0217] It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, an optical fiber, an RF (radio frequency) or the like, or any suitable combination of the above.

[0218] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, can implement the steps and corresponding contents of the foregoing method embodiments. Compared with related technologies, the following advantages can be achieved:

[0219] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described.

[0220] It should be understood that although each operation step in the flowchart of the embodiments of the present application is indicated by an arrow, the implementation order of the steps is not limited to the order indicated by the arrow. Unless explicitly stated herein, in some implementation scenarios of the embodiments of the present application, the implementation steps in each flowchart can be executed in other orders as required. In addition, part or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on the actual implementation scenario. Part or all of these sub-steps or stages can be executed at the same time, and each of these sub-steps or stages can also be executed at different times. In the scenario where the execution times are different, the execution order of these sub-steps or stages can be flexibly configured as required, and the embodiments of the present application do not limit this.

[0221] The above is only an optional implementation of some implementation scenarios of the present application. It should be pointed out that, for ordinary skilled persons in the technical field, other similar implementation means based on the technical idea of the present application without departing from the technical concept of the present application also belong to the protection scope of the embodiments of the present application.

Claims

1. A model training method, executed in an electronic device, comprising: obtaining at least one training sample, each training sample comprising a sample image of a human body and annotation information, the annotation information comprising three-dimensional actual coordinates of at least one key point of the human body in the sample image on a preset three-dimensional coordinate system; performing at least one round of training on a three-dimensional pose estimation model according to the at least one training sample to obtain a trained three-dimensional pose estimation model; wherein, for each training sample, each round of training comprises: inputting the sample image of the training sample into the current three-dimensional pose estimation model to obtain a three-dimensional Gaussian mixture representation of each key point on the preset three-dimensional coordinate system, the three-dimensional Gaussian mixture representation being used to represent a probability distribution of three-dimensional coordinates of the corresponding key point on the preset three-dimensional coordinate system on the sample image; for each key point, obtaining a two-dimensional Gaussian mixture representation of the key point on three mutually perpendicular two-dimensional planes in the preset three-dimensional coordinate system according to the three-dimensional Gaussian mixture representation of the key point, the two-dimensional Gaussian mixture representation being used to represent a probability distribution of two-dimensional coordinates of the corresponding key point on the corresponding two-dimensional plane; for each key point, determining a two-dimensional Gaussian distribution of the two-dimensional coordinates of the key point on the corresponding two-dimensional plane according to the two-dimensional actual coordinates of the key point on each two-dimensional plane and a preset standard deviation; for each key point, determining a loss value of the key point on each two-dimensional plane according to a difference between the two-dimensional Gaussian mixture representation and the two-dimensional Gaussian distribution of the key point on each two-dimensional plane; adjusting model parameters of the current three-dimensional pose estimation model based on the loss values of the respective key points on each two-dimensional plane.

2. The method of claim 1, wherein the determining, for each key point, a loss value of the key point on each two-dimensional plane according to a difference between the two-dimensional Gaussian mixture representation and the two-dimensional Gaussian distribution of the key point on each two-dimensional plane comprises: for each key point, taking a KL divergence loss between the two-dimensional Gaussian mixture representation and the corresponding two-dimensional Gaussian distribution of the key point on each two-dimensional plane as a first loss of the key point on the corresponding two-dimensional plane; for each key point, taking an L1 loss between the two-dimensional Gaussian mixture representation and the corresponding two-dimensional Gaussian distribution of the key point on each two-dimensional plane as a second loss of the key point on the corresponding two-dimensional plane; for each key point, determining the loss value of the key point on the corresponding two-dimensional plane according to the first loss and the second loss of the key point on each two-dimensional plane.

3. The method of claim 2, wherein the determining, for each key point, the loss value of the key point on the corresponding two-dimensional plane according to the first loss and the second loss of the key point on each two-dimensional plane comprises: for each key point, smoothing the second loss according to a preset smoothing coefficient to obtain a third loss; for each key point, taking a sum of the third loss and the first loss as the loss value of the key point on the corresponding two-dimensional plane.

4. The method of any one of claims 1-3, wherein the inputting of the sample image of the training sample into the current three-dimensional pose estimation model to obtain a three-dimensional Gaussian mixture representation of each key point in the preset three-dimensional coordinate system comprises: inputting the sample image into the current three-dimensional pose estimation model to obtain a three-dimensional predicted coordinate of each key point in the sample image generated by the three-dimensional pose estimation model, and a three-dimensional covariance matrix of a plurality of three-dimensional Gaussian distribution components and a weight of each three-dimensional Gaussian distribution component in the three-dimensional Gaussian mixture representation; determining, for each key point, the three-dimensional Gaussian mixture representation of the key point according to the three-dimensional predicted coordinate of the key point, the three-dimensional covariance matrix of the plurality of three-dimensional Gaussian distribution components, and the weight of each three-dimensional Gaussian distribution component.

5. The method of claim 4, wherein the determining, for each key point, the three-dimensional Gaussian mixture representation of the key point according to the three-dimensional predicted coordinate of the key point, the three-dimensional covariance matrix of the plurality of three-dimensional Gaussian distribution components, and the weight of each three-dimensional Gaussian distribution component comprises: determining, for each key point, a plurality of three-dimensional Gaussian distribution components of the key point according to the three-dimensional predicted coordinate of the key point and the three-dimensional covariance matrix of the plurality of three-dimensional Gaussian distribution components; and determining, for each key point, the three-dimensional Gaussian mixture representation of the key point according to the plurality of three-dimensional Gaussian distribution components of the key point and the weight of each three-dimensional Gaussian distribution component.

6. The method of claim 4 or 5, wherein the three-dimensional covariance matrix of the plurality of three-dimensional Gaussian distribution components of each key point is determined by: obtaining, for each key point, a standard deviation of each three-dimensional Gaussian distribution component of the key point in the preset three-dimensional coordinate system on three coordinate axes and a Pearson correlation coefficient between the standard deviations on two coordinate axes for each three-dimensional Gaussian distribution component of the key point generated by the three-dimensional pose estimation model; and determining, for each key point, a covariance matrix of each three-dimensional Gaussian distribution component of the key point according to the standard deviation of each three-dimensional Gaussian distribution component of the key point on the three coordinate axes and the Pearson correlation coefficient between the standard deviations on the two coordinate axes.

7. The method of any one of claims 1-6, wherein the obtaining, for each key point, a two-dimensional Gaussian mixture representation of the key point in three mutually perpendicular two-dimensional planes in the preset three-dimensional coordinate system according to the three-dimensional Gaussian mixture representation of the key point comprises: obtaining, for each key point, a two-dimensional predicted coordinate of the three-dimensional predicted coordinate of the key point in the three two-dimensional planes according to the three-dimensional predicted coordinate of the key point; and obtaining, for each key point, a three-dimensional covariance matrix of the plurality of three-dimensional Gaussian distribution components of the key point. ​ ​ ​ ​ ​ ​ wherein, ​ ​ ​ ​ For each three-dimensional Gaussian distribution component of the key point, determining a two-dimensional covariance matrix of the three-dimensional covariance matrix on each two-dimensional plane according to the three-dimensional covariance matrix of the three-dimensional Gaussian distribution component; For each key point, determining a plurality of two-dimensional Gaussian distribution components of the key point on each two-dimensional plane according to the two-dimensional predicted coordinates of the key point on each two-dimensional plane and the plurality of two-dimensional covariance matrices on the corresponding two-dimensional plane; For each key point, determining a two-dimensional Gaussian mixture representation of each key point on each two-dimensional plane according to the plurality of two-dimensional Gaussian distribution components of the key point on each two-dimensional plane and the weight of each two-dimensional Gaussian distribution component, the weight of each two-dimensional Gaussian distribution component being the same as the weight of the corresponding three-dimensional Gaussian distribution component.

8. The method of any one of claims 1-7, further comprising: obtaining a plurality of images; performing human body detection on the plurality of images to determine a region containing a human body in each of the plurality of images; determining three-dimensional actual coordinates of each key point of the human body in a preset three-dimensional coordinate system; cropping the region containing the human body from the image to obtain a sample image of a training sample; taking the three-dimensional actual coordinates of each key point of the human body in the preset three-dimensional coordinate system as annotation information of the corresponding training sample.

9. A three-dimensional pose estimation method, implemented in an electronic device, the method comprising: obtaining an image; determining a region containing a human body in the image by performing human body detection on the image; cropping the region containing the human body from the image to obtain a target image; determining three-dimensional coordinates of at least one key point from the target image based on a trained three-dimensional pose estimation model, wherein the three-dimensional pose estimation model is trained based on the method of any one of claims 1 to 8.

10. A model training apparatus, comprising: a first obtaining module configured to obtain at least one training sample, each training sample comprising a sample image of a human body and annotation information, the annotation information comprising three-dimensional actual coordinates of at least one key point of the human body in a preset three-dimensional coordinate system; a first training module configured to perform at least one round of training on a three-dimensional pose estimation model based on the at least one training sample to obtain a trained three-dimensional pose estimation model; wherein, for each training sample, each round of training comprises: inputting the sample image of the training sample into the current three-dimensional pose estimation model to obtain a three-dimensional Gaussian mixture representation of each key point in the preset three-dimensional coordinate system, the three-dimensional Gaussian mixture representation being used to represent a probability distribution of three-dimensional coordinates of the corresponding key point in the preset three-dimensional coordinate system on the sample image; for each key point, obtaining a two-dimensional Gaussian mixture representation of the key point on three mutually perpendicular two-dimensional planes in the preset three-dimensional coordinate system according to the three-dimensional Gaussian mixture representation of the key point, the two-dimensional Gaussian mixture representation being used to represent a probability distribution of two-dimensional coordinates of the corresponding key point on the corresponding two-dimensional plane; and For each key point, a two-dimensional Gaussian distribution of the two-dimensional coordinates of the key point on the corresponding two-dimensional plane is determined according to the two-dimensional actual coordinates of the key point on each two-dimensional plane and a preset standard deviation; For each key point, a loss value of the key point on each two-dimensional plane is determined according to a difference between the two-dimensional Gaussian mixture representation and the two-dimensional Gaussian distribution of the key point on each two-dimensional plane; Based on the loss values of the key points on each two-dimensional plane, the model parameters of the current three-dimensional pose estimation model are adjusted. 11.A three-dimensional pose estimation apparatus, comprising: a first detection module configured to acquire an image; perform human body detection on the image to determine a region containing a human body in the image; a second acquisition module configured to crop the region containing the human body from the image to obtain a target image; a third acquisition module configured to determine three-dimensional coordinates of at least one key point from the target image based on a trained three-dimensional pose estimation model, wherein the three-dimensional pose estimation model is trained based on the method of any one of claims 1 to 8. 12.An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the method of any one of claims 1 to 9. 13.A computer readable storage medium having stored thereon a computer program, wherein the computer program is executed by a processor to implement the steps of the method of any one of claims 1 to 9. 14.A computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the steps of the method of any one of claims 1 to 9.

Images