Model training method, human body posture detection method, apparatus, device and storage medium

Abstract

A model training method, a human body pose detection method, an apparatus, a device, and a medium are disclosed, which include: inputting an image training sample corresponding to a current iteration into a current pose detection network model, obtaining a first loss function corresponding to the current iteration based on a data format of the image training sample, performing a reprojection process on the current output result of the pose detection network model based on the obtained camera parameters, obtaining a second loss function corresponding to the current iteration based on the result of the reprojection process, and backpropagating the pose detection network model based on a fitting loss function formed by the first loss function and the second loss function, to obtain a pose detection network model for a next iteration.

Description

[Technical field]

[0001] INCORPORATION OF RELATED APPLICATIONS This application claims priority to a Chinese patent application with application number 202011176560.4 filed with the China Patent Office on October 28, 2020, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of spatial position detection technology, for example, to a model training method, a human body pose detection method, an apparatus, a device and a medium. [Background technology]

[0003] With the rapid development of artificial intelligence technology, artificial neural networks have been widely applied. Artificial neural networks, also known as neural networks, are models that process information by simulating the structure of synaptic connections in the brain. In the field of 3D spatial position detection, neural network technology can be used to build a posture estimation neural network model for detecting 3D posture of the human body.

[0004] Studying 3D human body pose detection has become an important problem in the field of computer vision. Such technology is considered to be the basis for computers to understand human behavior. It has a wide range of future applications in fields such as human-computer interaction, film and television special effects, and interactive entertainment. In general, human body pose detection corresponds to detecting human body keypoints from images containing a human body and obtaining the 3D spatial position coordinates of the human body keypoints.

[0005] However, the posture detection neural network model for detecting the 3D posture of the human body has the following problems: Problem 1: The industry lacks effective natural scene 3D human body labeling means, so there is a lack of high-quality unconstrained scene label data; Problem 2: The model has a problem of being difficult to converge, so it is not possible to realize accurate detection of the 3D posture of the human body in natural scenes based on the posture detection neural network model. Summary of the Invention [Means for solving the problem]

[0006] The present application provides a model training method, a human body pose detection method, an apparatus, a device and a medium, which realizes effective training of a network model required for human body pose detection, and ensures the accuracy of the human body pose detection result.

[0007] The present application provides a method for inputting an image training sample corresponding to a current iteration into a current pose detection network model, and obtaining a first loss function corresponding to the current iteration according to a data format of the image training sample; According to the obtained camera parameters, perform a reprojection process on the current output result of the pose detection network model, and obtain a second loss function corresponding to the current iteration according to the result of the reprojection process; Backpropagating the posture detection network model according to a fitting loss function formed by the first loss function and the second loss function to obtain a posture detection network model for a next iteration, and entering the next iteration until an iteration termination condition is met, thereby realizing training for the posture detection network model; Including, Regarding model training methods.

[0008] The present application provides a method for detecting a person by acquiring an actual scene image of the person to be detected; The person real scene image is inputted into a preset pose detection network model, where the preset pose detection network model is obtained by training using the method according to the above embodiment; performing human body posture detection for the person in the actual person scene image based on an output result of the preset posture detection network model, thereby obtaining three-dimensional spatial position coordinates of a plurality of key points of the person; Including, The present invention relates to a method for detecting a human body posture.

[0009] The present application provides a method for detecting a pose of a subject, comprising: a first information determination module configured to input an image training sample corresponding to a current iteration into a current pose detection network model, and obtain a first loss function corresponding to the current iteration based on a data format of the image training sample; A second information determination module configured to perform a reprojection process on a current output result of the pose detection network model based on the obtained camera parameters, and obtain a second loss function corresponding to the current iteration based on a result of the reprojection process; a model training module configured to backpropagate the posture detection network model according to a fitting loss function formed by the first loss function and the second loss function to obtain a posture detection network model for a next iteration, and enter the next iteration until an iteration termination condition is met, thereby realizing training for the posture detection network model; Including, Concerning model training devices.

[0010] The present application relates to an information acquisition module configured to acquire a real scene image of a person to be detected; An information input module configured to input the person real scene image to a preset pose detection network model, where the preset pose detection network model is obtained by training according to the method according to any one of claims 1 to 10; an information determination module configured to perform human body pose detection on the person in the real person scene image based on the output result of the preset pose detection network model, thereby obtaining three-dimensional spatial position coordinates of a plurality of key points of the person; Including, This invention relates to a human body posture detection device.

[0011] The present application relates to a computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, which, when executed by the processor, implements the above method.

[0012] The present application relates to a computer-readable storage medium on which a computer program is stored, the program implementing the above method when executed by a processor. [Brief description of the drawings]

[0013] [Figure 1] 1 is a flowchart of a model training method according to Example 1 of the present application. [Diagram 2] 1 is a flowchart of a model training method according to Example 2 of the present application. [Figure 2a] FIG. 11 is a schematic diagram of a structure for training a posture detection network model according to Example 2 of the present application. [Diagram 3] 11 is a flowchart of a human body posture detection method according to a third embodiment of the present application. [Figure 4] FIG. 11 is a block diagram of a configuration of a model training device according to a fourth embodiment of the present application. [Diagram 5] FIG. 11 is a block diagram of a configuration of a human body posture detecting device according to a fifth embodiment of the present application. [Figure 6] FIG. 13 is a block diagram of a configuration of a computer device according to a sixth embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The present application will now be described with reference to the drawings and examples. The specific examples described in this specification are only used to explain the present application. For ease of explanation, only those relevant to the present application are shown in the drawings.

[0015] Before discussing the exemplary embodiments, it should be mentioned that some exemplary embodiments are described as a process or method that is depicted as a flowchart. In the flowchart, multiple steps are described as a sequential process, but many of them may be performed in parallel, synchronously, or simultaneously. Also, the order of multiple steps can be rearranged. A process may be terminated when its operations are completed, but may have additional steps not included in the drawing. The process may correspond to a method, a function, a rule, a subroutine, a subprogram, etc.

[0016] Example 1 FIG. 1 is a flowchart of a model training method according to Example 1 of the present application, which may be performed by a model training apparatus, where the apparatus may be realized by software and / or hardware and is generally integrated into a computing device.

[0017] In the neural network model used for human body pose detection, coordinate regression network model and heat map regression network model are two commonly seen network models. In the case of the coordinate regression network model, the structure of the model is usually composed of a front end of a convolutional network and a back end of a fully connected network. In the case of human body pose detection using this network model, the 3D coordinates of the human body keypoints can be directly output. However, such a network model has the following disadvantages: it is difficult to converge, and it is difficult to obtain 3D label data in natural scenes. In the case of the heat map regression network model, the structure of such a model is usually composed of only a fully convolutional network. The output result is a coordinate heat map of the human body keypoints. After that, the 3D coordinates of the keypoints need to be obtained by analyzing the heat map. Such a network model has the following disadvantages: it has high requirements for training samples, and it is difficult to use 2D label data in unconstrained scenes. Similarly, it is difficult to obtain 3D label data in natural scenes. As a result, it is difficult to apply the network to real scenes. Therefore, in the embodiments of the present application, the following model training method is provided, that is, by utilizing the network structure of the heat map regression network, and through reprojection in the image domain and easily obtained image training samples, the problems existing in the above network model training can be effectively solved.

[0018] As shown in FIG. 1, the model training method according to the first embodiment includes the following steps:

[0019] In S101, an image training sample corresponding to a current iteration is input into a current pose detection network model, and a first loss function corresponding to the current iteration is obtained based on a data format of the image training sample.

[0020] In this embodiment, the image training sample is a data pair of "sample image-coordinate label" collected in advance, where the sample image may be a monocular image including a person captured by a monocular camera, the coordinate label may be the keypoint coordinate of the person included in the sample image, and the keypoint coordinate may be the two-dimensional plane coordinate of the keypoint in the sample image. If the sample image is a specific image collected in a constrained scene, the keypoint coordinate may be the three-dimensional spatial coordinate of the keypoint in the constrained scene.

[0021] A monocular camera is a common image collection device that has the characteristics of having a wide range of applications and being easily available. It is possible to obtain sample images in the image training sample of this embodiment using only this monocular camera, which better embodies the ease of constructing and collecting the data necessary for training in this embodiment.

[0022] At the same time, the current iteration can be understood as realizing one of the iterative operations during model training, and in this embodiment, in the iterative training of the model, the operational steps of performing model training are considered to be the same in each iteration.

[0023] In this step, after entering the current iteration, select an image training sample to participate in model training, and the image training sample can be selected from a training sample set collected in advance.Then, the selected image training sample can be input to the posture detection network model that needs to be trained in the current iteration, where the posture detection network model used in this embodiment can be a heat map regression network model constructed using the structure of a fully convolutional network.The posture detection network model can process the input image training sample, and at the same time, can obtain a first loss function that is required for model training in the current iteration based on the data format of the image training sample.

[0024] The data format of the image training sample in this embodiment may be a natural scene sample image-keypoint two-dimensional coordinate label format, or a constrained scene sample image-keypoint two-dimensional coordinate and keypoint three-dimensional coordinate label format, that is, one is that the sample image is collected in a natural scene, and the corresponding label is the keypoint two-dimensional coordinate; the other is that the sample image is collected in a constrained scene, and the corresponding label is the keypoint two-dimensional label and the keypoint three-dimensional coordinate.

[0025] The data information contained in the image training samples with different data formats is different, and in this embodiment, the data information contained in the image training samples can determine which first loss function is used for subsequent training. For example, when the image training samples only contain two-dimensional coordinates of key points, the only thing that can be considered is to combine the data information corresponding to the two-dimensional coordinates of key points with the output result of the network model to form the first loss function, and when participating in reverse learning through the first loss function obtained at this time, it is guaranteed that the posture detection network model can effectively learn the image features of natural scenes.

[0026] For example, if the image training sample also contains the three-dimensional coordinates of the keypoints, the data information corresponding to the two-dimensional coordinates of the keypoints and the three-dimensional coordinates of the keypoints can be combined with the output result of the network model to form a first loss function. When the first loss function obtained at this time is used to participate in the inverse learning of the model, it is guaranteed that the pose detection network model can effectively learn the accurate three-dimensional pose prediction ability.

[0027] In S102, a reprojection process is performed on the current output result of the pose detection network model based on the obtained camera parameters, and a second loss function corresponding to the current iteration is obtained based on the result of the reprojection process.

[0028] In this embodiment, multiple loss functions can be combined to train the posture detection network model, thereby achieving the effect of comprehensively training the characteristics. Therefore, after determining the first loss function in S101, this embodiment needs to further determine the second loss function in S102.

[0029] After processing the image training sample with the pose detection network model, a current output result can be obtained, which includes a set of heat maps required to determine the 3D space coordinates for each keypoint. In this step, the current output result can be reprojected into 2D space according to the camera parameters on which the sample images in the image training sample are collected, thereby obtaining a set of heat maps of the keypoints in 2D space.

[0030] In this step, the reprojection result after performing the reprojection process on the current output result can be combined with the corresponding standard value set in the two-dimensional space of the sample image in the image training sample to obtain the second loss function in the current iteration. Here, for the corresponding standard value set in the two-dimensional space of the sample image, if the data format of the image training sample is the label of the keypoint three-dimensional coordinates, it can be obtained by first performing the reprojection of the keypoint three-dimensional coordinates and then performing the heat map transformation; if the data format of the image training sample is the label of the keypoint two-dimensional coordinates, it can be directly performed the heat map transformation of the keypoint two-dimensional coordinates.

[0031] In S103, the posture detection network model is backpropagated based on a fitting loss function formed by the first loss function and the second loss function to obtain a posture detection network model for the next iteration, and the next iteration is set as a new current iteration and the process returns to S101 until an iteration termination condition is met.

[0032] In this embodiment, the first loss function and the second loss function can be directly integrated to form a fitting loss function, and the present embodiment can adopt an integrated form in which the fitting loss function is determined as the sum of the first loss function and the second loss function.

[0033] In training a neural network model, the backpropagation method can continuously update and adjust the weights (also called filters) of the network until the output of the network matches the target, and is a method for effectively calculating the gradient. In the embodiment of the present application, after determining the fitting loss function corresponding to the current iteration, the currently adopted posture detection network model is backpropagated using the fitting loss function, and the posture detection network model after the network weights are adjusted can be obtained, and the adjusted posture detection network model can be used to train the model in the next iteration. The embodiment of the present application does not limit the process of backpropagation, and can be set according to circumstances.

[0034] After the above S101 to S103 are performed, it corresponds to the completion of model training in one iteration. In this case, it can be determined whether the iteration end condition is met after the current iteration. If the iteration end condition is not met, the next iteration can be the new current iteration and can be executed again by returning to the above S101. If the iteration end condition is met, it corresponds to the completion of model training, and the model training method according to this embodiment can be terminated. Here, the iteration end condition can be that all the image training samples in the sample set participate in model training, or that the change in the network weight of the network model after training in the current iteration and the network model after training in the previous iteration is within a reasonable error range.

[0035] The model training method according to the first embodiment of the present application can perform model training only with natural scene images with 2D labels or constrained scene images with 2D labels and 3D labels without specifying image training samples, reducing the difficulty of collecting training samples, reducing the domain difference between training data and actual application scenes, and ensuring the realization of network training. At the same time, compared with the conventional reprojection process in the coordinate domain, the above technical solution can simply and effectively realize reprojection in the image domain, complete the entire model training in the image domain, and is favorable for the stability and rapid convergence of model training. When using the model trained in this way to perform human body pose detection, the 3D space coordinates of the human key points in the human image can be more accurately detected, and the accuracy of human body 3D pose detection can be effectively improved.

[0036] Example 2 FIG. 2 is a flowchart of a model training method according to Example 2 of the present application. Example 2 is described based on the above example. In this example, obtaining a first loss function corresponding to the current iteration based on the data format of the image training sample includes: extracting a first current probability heat map from the current output result of the pose detection network model; obtaining a first standard probability heat map determined in advance; and obtaining a first loss function corresponding to the current iteration based on the first standard probability heat map and the first current probability heat map, where the first standard probability heat map is determined by transforming two-dimensional coordinates of keypoints in the image training sample.

[0037] Alternatively, obtaining a first loss function corresponding to the current iteration based on the data format of the image training samples includes: when the data format of the image training samples is a label format of a constrained scene sample image-keypoint 2D coordinates and keypoint 3D coordinates, obtaining a current heatmap set output from the pose detection network model, where the current heatmap set includes a second current probability heatmap and a group of current 3D coordinate heatmaps; obtaining a pre-determined standard heatmap set, and obtaining a first loss function corresponding to the current iteration based on the standard heatmap set and the current heatmap set, where the standard heatmap set is determined by a transformation on the keypoint 3D coordinates in the image training samples, and the standard heatmap set includes a second standard probability heatmap and a group of standard 3D coordinate heatmaps.

[0038] At the same time, in this embodiment, performing a reprojection process on the current output result of the pose detection network model based on the obtained camera parameters includes: obtaining camera parameters contained in the image training sample, or obtaining camera parameters output from one output branch of the pose detection network model corresponding to the image training sample; extracting a current 3D abscissa heat map and a current 3D ordinate heat map from the current output result of the pose detection network model; combining a set reprojection formula based on the camera parameters, the current 3D abscissa heat map and the current 3D ordinate heat map to determine a current 2D abscissa heat map and a current 2D ordinate heat map of key points in the image training sample; and taking the current 2D abscissa heat map group formed by the current 2D abscissa heat map and the current 2D ordinate heat map as the result of the reprojection process.

[0039] In addition to the above technical solution, in this embodiment 2, obtaining a second loss function corresponding to the current iteration based on the result of the reprojection process includes: obtaining a pre-determined group of standard 2D coordinate heat maps, where the standard 2D coordinate heat maps are determined by transforming the 2D coordinates of key points or the 3D coordinates of key points in the image training sample, and include a standard 2D abscissa heat map and a standard 2D ordinate heat map; determining a fourth difference formula between the data corresponding to the standard 2D coordinate heat maps and the data corresponding to the current 2D abscissa heat maps; and determining a fourth squared formula of the Euclidean norm formed based on the fourth difference formula as the second loss function corresponding to the current iteration.

[0040] As shown in FIG. 2, the model training method according to Example 2 of the present application includes the following operations.

[0041] In S201, the image training sample corresponding to the current iteration is input into the current pose detection network model.

[0042] In S202, the data format of the image training samples is determined.

[0043] In this embodiment, the data format of an image training sample can be determined according to the data information actually contained in the image training sample. For example, if it is assumed that the image training sample contains data information of a sample image, a keypoint two-dimensional coordinate, and a keypoint three-dimensional coordinate, the data format of the image training sample can be considered to be a label format of a constrained scene sample image-a keypoint two-dimensional coordinate and a keypoint three-dimensional coordinate; for example, if it is assumed that the image training sample only contains data information of a sample image and a keypoint two-dimensional coordinate, the data format can be considered to be a label format of a natural scene sample image-a keypoint two-dimensional coordinate.

[0044] In the following S203 and S204 of this embodiment, a process of determining a first loss function is provided when the data format is a label format of natural scene sample image-keypoint two-dimensional coordinates.

[0045] In S203, if the data format is a natural scene sample image-keypoint two-dimensional coordinate label format, a first current probability heat map is extracted from the current output result of the pose detection network model.

[0046] In this embodiment, regardless of the data format of the image training sample, the current output result obtained by the pose detection network model is a heatmap set corresponding to each keypoint, and the corresponding heatmap set actually includes the probability heatmap of the keypoint and a group of 3D coordinate heatmaps representing the 3D spatial coordinates. In this step, the probability heatmap corresponding to each keypoint is extracted from the obtained current output result of the pose detection network model, which can be denoted as the first current probability heatmap.

[0047] In S204, a first standard probability heat map determined in advance is obtained, and a first loss function corresponding to the current iteration is obtained based on the first standard probability heat map and the first current probability heat map.

[0048] In this embodiment, the first standard probability heat map is regarded as the probability heat map that the sample image in the image training sample really has, and the first standard probability heat map is determined by transformation to the keypoint two-dimensional coordinate in the image training sample. Illustratively, when the two-dimensional coordinate of the keypoint is known, the probability heat map in the image domain can be obtained by the two-dimensional coordinate in the coordinate domain.

[0049] The formation of a loss function in model training relies on a comparison between the results output from the model and the marked-up actual results, and in this step, a first loss function corresponding to the current iteration can be obtained by comparing a first standard probability heatmap with data corresponding to the first current probability heatmap.

[0050] In this embodiment, obtaining a first loss function corresponding to the current iteration based on the first standard probability heat map and the first current probability heat map includes determining a first difference equation between data corresponding to the first standard probability heat map and data corresponding to the first current probability heat map, and determining a first squared equation of Euclidean norm formed based on the first difference equation as a first loss function corresponding to the current iteration.

[0051] In this embodiment, each heat map is considered as a graph matrix in which each of the pixel points contained therein has a corresponding data value, and in this embodiment, the graph matrix can be considered as data corresponding to the heat map, such that a first difference equation between the data corresponding to the first standard probability heat map and the data corresponding to the first current probability heat map can be considered as a matrix difference between the graph matrix corresponding to the first standard probability heat map and the graph matrix corresponding to the first current probability heat map.

[0052] For example,

number

number

number

number

number

number

[0053] In the following S205 and S206 of this embodiment, a process of determining a first loss function is provided when the data format is a label format of a constrained scene sample image-keypoint two-dimensional coordinates and keypoint three-dimensional coordinates.

[0054] In S205, if the data format is a label format of a constraint scene sample image-keypoint two-dimensional coordinates and keypoint three-dimensional coordinates, a current heat map set output from the pose detection network model is obtained.

[0055] In this embodiment, the current output result obtained by the posture detection network model processing the image training sample actually includes a heatmap set corresponding to each keypoint, and the heatmap set described in this embodiment is the current heatmap set. For ease of distinction, in this step, the probability heatmap included in the current heatmap set corresponding to the keypoint is referred to as the second current probability heatmap, and the included 3D coordinate heatmap group is referred to as the current 3D coordinate heatmap group.

[0056] In S206, a pre-established standard heatmap set is obtained, and a first loss function corresponding to the current iteration is obtained based on the standard heatmap set and the current heatmap set.

[0057] In this embodiment, the standard heatmap set is considered as a heatmap set that the sample image in the image training sample actually possesses, and the standard heatmap set is determined by transformation of the two-dimensional coordinates and three-dimensional coordinates of the keypoints contained in the image training sample. For example, when the two-dimensional coordinates of the keypoints are known, a probability heatmap in the image domain can be obtained by the two-dimensional coordinates in the coordinate domain; at the same time, when the three-dimensional coordinates of the keypoints are known, a coordinate heatmap corresponding to each of the three coordinate directions of the three-dimensional coordinates in the image domain can be obtained by the three-dimensional coordinates in the coordinate domain, and the coordinate heatmaps of the three coordinate directions form a three-dimensional coordinate heatmap group.

[0058] In this embodiment, the obtained probability heatmap is referred to as the second label probability heatmap, and the 3D coordinate heatmap group is referred to as the standard 3D coordinate heatmap group. In addition, when the 3D coordinate heatmap group is known, the 2D coordinate heatmap group of the image training sample can also be obtained by reprojection.

[0059] Similarly, the image training sample corresponds to the first loss function determined by the current iteration in the above data format, and also depends on the comparison between the actual result and the output result of the image training sample. Illustratively, in this step, the data corresponding to the second standard probability heatmap and the standard 3D coordinate heatmap group can be compared with the data corresponding to the second current probability heatmap and the current 3D coordinate heatmap group, respectively, to obtain the first loss function corresponding to the current iteration.

[0060] In this embodiment, obtaining a first loss function corresponding to the current iteration based on the standard heatmap set and the current heatmap set includes: determining a second difference formula between data corresponding to the second standard probability heatmap and data corresponding to the second current probability heatmap; determining a third difference formula between data corresponding to the standard 3D coordinate heatmap group and data corresponding to the current 3D coordinate heatmap group; forming a second squared formula of Euclidean norm based on the second difference formula, forming a third squared formula of Euclidean norm based on the third difference formula, and adding the second squared formula and the third squared formula to construct a first loss function corresponding to the current iteration.

[0061] Similarly, in this step, the data corresponding to the second standard probability heat map and the second current probability heat map are both graph matrices, and the data corresponding to the standard 3D coordinate heat map group and the current 3D coordinate heat map group can each be composed of three graph matrices.

[0062] For example,

number

number

number

number

number

number

number

[0063] In the following steps S207 to S210 of this embodiment, an implementation process for performing reprojection processing on the current output result is provided.

[0064] In S207, camera parameters included in the image training sample are obtained, or camera parameters output from one output branch of the pose detection network model corresponding to the image training sample are obtained.

[0065] In this embodiment, the camera parameters of the sample image may be directly included in the formed image training sample, in which case, this step can directly obtain the camera parameters included in the image training sample. In addition, the camera parameters may not be directly present in the image training sample, but can add another output branch from the pose detection network model, and obtain the camera parameters extracted after model processing from the output branch.

[0066] In S208, a current 3D abscissa heat map and a current 3D ordinate heat map are extracted from the current output result of the pose detection network model.

[0067] After the pose detection network model processes the sample image, the current output result includes a probability heat map and a group of three-dimensional coordinate heat maps corresponding to the key points. In this embodiment, it is considered that the group of three-dimensional coordinate heat maps of the current output result includes a current three-dimensional abscissa heat map and a current three-dimensional ordinate heat map. Thus, the current three-dimensional abscissa heat map and the current three-dimensional ordinate heat map included in the current output result can be obtained by this step.

[0068] In S209, based on the camera parameters, the current 3D abscissa heat map and the current 3D ordinate heat map, the set reprojection formula is combined to determine the current 2D abscissa heat map and the current 2D ordinate heat map of keypoints in the image training sample.

[0069] In this embodiment, the camera parameters are focal length parameters f x , f y , and the principal optical axis parameter c x , c y Now the three-dimensional abscissa heat map is represented by the corresponding graph matrix P 3d (x), and now the three-dimensional abscissa heat map can be represented by the corresponding graph matrix P 3d (y), and the reprojection formula can be expressed as follows: P 2d (x)=f x P 3d (x)+c x , P 2d (y)=f y P 3d (y)+c y

[0070] Here, P 2d (x) and P 2d (y) are represented as the current 2D abscissa heatmap and the current 2D ordinate heatmap of corresponding keypoints in the image training sample, respectively.

[0071] In S210, the current two-dimensional abscissa heat map and the current two-dimensional ordinate heat map form a group of current two-dimensional abscissa heat maps as a result of a reprojection process.

[0072] In this step, the current two-dimensional abscissa heat maps formed based on the current two-dimensional abscissa heat map and the current two-dimensional ordinate heat map can be regarded as a superimposition processing result of the reprojection processing.

[0073] In the following steps S211 to S213 of this embodiment, a process for determining the second loss function is provided.

[0074] In S211, a group of standard two-dimensional coordinate heat maps that have been determined in advance is obtained.

[0075] In this embodiment, the group of standard 2D coordinate heatmaps is determined by transformation to the 2D coordinates of keypoints in the image training samples, and the group of standard 2D coordinate heatmaps includes a standard 2D abscissa heatmap and a standard 2D ordinate heatmap.

[0076] In S212, a fourth difference equation between the data corresponding to the standard two-dimensional coordinate heat map group and the data corresponding to the current two-dimensional abscissa coordinate heat map group is determined.

[0077] In this embodiment, the data corresponding to the standard two-dimensional coordinate heat map group is a graph matrix of the standard two-dimensional coordinate heat map group.

number

number

[0078] In S213, a fourth squared formula of the Euclidean norm formed based on the fourth difference formula is determined as a second loss function corresponding to the current iteration.

[0079] In this embodiment, the fourth square equation of the Euclidean norm formed based on the fourth difference equation is:

number

number

[0080] In S214, the posture detection network model is backpropagated based on a fitting loss function formed by the first loss function and the second loss function to obtain a posture detection network model for the next iteration, and the next iteration is set as a new current iteration and the process returns to S201 until an iteration termination condition is met.

[0081] In this embodiment, the fitting loss function can be expressed as Loss=Loss1+Loss2.

[0082] To facilitate understanding of the implementation of the model training method according to this embodiment, Figure 2a is a schematic diagram of a structure for training a pose detection network model in an embodiment of the present application, in which, as shown in Figure 2a, a sample image 21 to be input, a pose detection network model 22, a first group of heatmaps 23 (including a probability heatmap and a three-dimensional coordinate heatmap group), camera parameters 24 and a second group of heatmaps 25 are shown.

[0083] The first heatmap group 23 is mainly output from the core output branch of the network model and is used to determine the 3D spatial coordinates of the human body keypoints, from which the contour map of the human body can be determined, as shown in Fig. 2a. The second heatmap group 25 is mainly obtained based on a reprojection process by combining the camera parameters 24 output from another output branch of the network model with the first heatmap group 23.

[0084] 2a shows a schematic diagram of model training, in which the adopted image training samples are mainly collected by a monocular camera, and at the same time, the first loss function Loss1 is determined mainly by the first heat map group 23 and the label data (key point two-dimensional coordinates or key point three-dimensional coordinates) corresponding to the sample image, and the second loss function Loss2 is determined mainly by the second heat map group 25 and the label data (key point two-dimensional coordinates or key point three-dimensional coordinates) corresponding to the sample image. As can be seen from this example, the model training method according to this embodiment has the characteristics that the training samples can be easily collected, and since training is performed only in the image domain, the training method also has the characteristics that the training is stable and converges quickly.

[0085] In the second embodiment of the present application, a model training method is provided, which describes the process of determining the first loss function and the process of determining the second loss function. The method according to this embodiment is equivalent to setting two loss functions for training learning for the posture detection network model, and the probability heat map, the group of 3D coordinate heat maps, and the group of 2D coordinate heat maps output from the posture detection network model are compared with the standard probability heat map, the group of standard 3D coordinate heat maps, and the group of standard 2D coordinate heat maps to determine the two loss functions. In the realization of this method, image data that can be easily collected at the same time is used as a sample image, and semi-supervised training can be performed in combination with reprojection, with good generalization, and only a fully convolutional structure in the image domain is adopted to ensure that the training process is entirely carried out in the image domain, so that factors affecting training such as gradient vanishing or gradient explosion during the training process can be avoided, and the objective of stable training and fast convergence is achieved. At the same time, when human body posture detection is performed using the model trained in this way, the 3D spatial coordinates of human key points in human images can be detected more accurately, and the accuracy of human body 3D posture detection can be effectively improved.

[0086] Example 3 3 is a flowchart of a human body posture detection method according to a third embodiment of the present application, the method being performed by a human body posture detection device, where the device can be realized by software and / or hardware, and is generally integrated into a computer device. As shown in FIG. 3, the method includes the following steps:

[0087] In S301, an actual image of a person to be detected is acquired.

[0088] Illustratively, the person real-scene image is captured by a conventional image capture device, and the image capture device may be a mobile phone, a laptop with a camera, or a tablet. When the human body posture needs to be detected, this step can first obtain the person real-scene image to be detected.

[0089] In S302, the real person image is input to a preset pose detection network model.

[0090] The preset pose detection network model is obtained by training using any model training method according to the embodiments of the present application.

[0091] In S303, based on the output result of the preset posture detection network model, human body posture detection is performed on the person in the actual person scene image, thereby obtaining three-dimensional spatial position coordinates of a plurality of key points of the person.

[0092] For example, the output result after processing by the posture detection network model is a group of heat maps corresponding to the key points, and for each key point, the obtained group of heat maps can be used to obtain the corresponding 3D spatial position coordinates by analyzing the heat maps in the group of heat maps.

[0093] In the human body posture detection method according to Example 3 of the present application, a posture detection network model obtained by training using the above-mentioned model training method is used to obtain a group of heat maps required for human body posture detection. The group of heat maps obtained by the posture detection network model can more accurately detect the three-dimensional spatial coordinates of human key points in a human image, and can effectively improve the accuracy of human body three-dimensional posture detection.

[0094] As an optional embodiment of Example 3 of the present application, the optimized output result includes a probability heat map and a group of 3D coordinate heat maps of multiple keypoints. Correspondingly, in this optional embodiment, based on the output result of the preset pose detection network model, performing human body pose detection on the person in the real-scene image of a person, thereby obtaining 3D spatial position coordinates of multiple keypoints of the person, is optimized as follows: for each keypoint, a probability coordinate point corresponding to the maximum probability value is determined from the probability heat map, current position coordinates of the probability coordinate point are obtained, and a numerical value corresponding to the current position coordinate is searched for in each of the 3D abscissa coordinate heat map, 3D ordinate coordinate heat map and 3D depth coordinate heat map included in the group of 3D coordinate maps, and the 3D spatial position coordinate of the keypoint is constructed based on the searched numerical value.

[0095] In this optional embodiment, the 3D spatial position coordinates of the keypoints are determined, and a corresponding group of heatmaps can be obtained for each keypoint whose 3D spatial position coordinates are to be determined by the pose detection network model, where the group of heatmaps includes a probability heatmap of the keypoint, which indicates the probability value when the keypoint is at different coordinates, and also includes a group of 3D coordinate heatmaps of the keypoint, where the group of 3D coordinate heatmaps includes a 3D horizontal coordinate heatmap, a 3D vertical coordinate heatmap and a 3D depth coordinate heatmap, where each coordinate heatmap indicates the coordinate value that the keypoint may have in the corresponding coordinate direction.

[0096] A maximum probability value can be determined from the probability heat map of the keypoint, where the maximum probability value corresponds to a probability coordinate point on the probability heat map, and the maximum probability value indicates that the keypoint has the maximum probability of being at the 3D spatial position determined based on the probability coordinate point. For a given group of 3D coordinate heat maps, numerical values ​​corresponding to the probability coordinate point can be searched for from different coordinate heat maps, and the determined numerical values ​​are used as coordinate values ​​in the corresponding coordinate directions, thereby forming the 3D spatial position coordinates of the keypoint using the three determined coordinate values.

[0097] For example, assuming that the probability coordinate point determined from the probability heat map is (x1, y1), a point with coordinate values ​​(x1, y1) can be found in each of the 3D abscissa coordinate heat map, the 3D ordinate coordinate heat map and the 3D depth coordinate heat map, and target values ​​corresponding to the point in different heat maps can be obtained, and finally, the 3D spatial position coordinates of the key point are obtained based on the three target values.

[0098] As another optional embodiment of this embodiment, the human body posture detection method includes adding 3D spatial position coordinates based on multiple key points of the person in the real person image, and converting the real person image into a corresponding cartoon person image.

[0099] In this optional embodiment, the determined 3D spatial position coordinates of the keypoints can be used to transform the real person image into a cartoon person image, which corresponds to providing a practical application scene of human body 3D posture detection. After the 3D spatial position coordinates of the keypoints in the real person image are determined, the original 3D spatial position coordinates of the keypoints in one reference cartoon person image are adjusted so that the keypoints have the determined 3D spatial position coordinates, which corresponds to realizing the adjustment of the human body position in the reference cartoon person image, and a cartoon person image corresponding to the real person image is formed.

[0100] Example 4 4 is a block diagram of a model training apparatus according to a fourth embodiment of the present application, which can be realized by software and / or hardware, and is generally integrated into a computer device, and can perform model training by executing a model training method. As shown in FIG. 4, the apparatus includes a first information determination module 41, a second information determination module 42 and a model training module 43.

[0101] The first information determination module 41 is configured to input an image training sample corresponding to the current iteration into a current posture detection network model, and obtain a first loss function corresponding to the current iteration based on a data format of the image training sample; the second information determination module 42 is configured to perform a reprojection process on the current output result of the posture detection network model based on the obtained camera parameters, and obtain a second loss function corresponding to the current iteration based on the result of the reprojection process; and the model training module 43 is configured to backpropagate the posture detection network model based on a fitting loss function formed by the first loss function and the second loss function, obtain a posture detection network model for the next iteration, and enter the next iteration until an iteration termination condition is met, thereby realizing training for the posture detection network model.

[0102] The model training device according to the fourth embodiment of the present application can perform model training only with natural scene images with 2D labels or constrained scene images with 2D and 3D labels without identifying image training samples, reducing the difficulty of collecting training samples, reducing the domain difference between training data and actual application scenes, and ensuring the realization of network training. At the same time, compared with the traditional reprojection process in the coordinate domain, the above technical solution can simply and effectively realize reprojection in the image domain, complete the entire model training in the image domain, and is favorable for the stability and rapid convergence of model training. When the model trained in this way is used to perform human body pose detection, the 3D space coordinates of the human key points in the human image can be more accurately detected, and the accuracy of human body 3D pose detection can be effectively improved.

[0103] Example 5 5 is a block diagram of a configuration of a human body posture detection device according to a fifth embodiment of the present application, the device can be realized by software and / or hardware, and is generally integrated into a computer device, and can realize the determination of the 3D spatial position of the human body keypoints by executing the human body posture detection method. As shown in FIG. 5, the device includes an information acquisition module 51, an information input module 52 and an information determination module 53.

[0104] The information acquisition module 51 is configured to acquire a real-world image of a person to be detected, the information input module 52 is configured to input the real-world image of a person to a preset pose detection network model, where the preset pose detection network model is obtained by training using a model training method according to an embodiment of the present application, and the information determination module 53 is configured to obtain three-dimensional spatial position coordinates of multiple key points of the person by performing human body pose detection on the person in the real-world image of a person based on the output result of the preset pose detection network model.

[0105] The human body posture detection device of this embodiment uses a posture detection network model obtained by training using the above-mentioned model training method to obtain a group of heat maps required for human body posture detection, and the group of heat maps obtained by the posture detection network model can more accurately detect the three-dimensional spatial coordinates of human key points in a human image, thereby effectively improving the accuracy of human body three-dimensional posture detection.

[0106] Example 6 FIG. 6 is a block diagram of a configuration of a computing device according to Example 6 of the present application. In Example 6 of the present application, a computing device is provided, in which a model training device and / or a human body posture detection device according to an example of the present application can be integrated.

[0107] As shown in FIG. 6, the computing device 700 includes a memory 701, a processor 702, and a computer program stored in the memory 701 and executable on the processor 702, and when the processor 702 executes the computer program, it realizes the model training method and / or the human body pose detection method according to an embodiment of the present application.

[0108] At the same time, Example 6 of the present application further provides a storage medium including computer-executable instructions, which, when executed by a processor of a computer, performs a model training method and / or a human body pose detection method according to an embodiment of the present application.

[0109] A storage medium is any of various types of memory or storage devices. The term "storage medium" includes installation media such as Compact Disc Read-Only Memory (CD-ROM), floppy disk or magnetic tape drives, computer system memory or random access memory such as Dynamic Random Access Memory (DRAM), Double Data Rate Random Access Memory (DDRRAM), Static Random Access Memory (SRAM), Extended Data Output Random Access Memory (EDORAM), Rambus RAM, non-volatile memory such as flash memory, magnetic media (e.g. hard disks and optical storage devices), registers and other similar types of memory elements, etc. A storage medium may be a non-transitory storage medium, and a storage medium may include other types of memory or combinations thereof. It should be noted that the storage medium may be located in the first computer system on which the program is executed, or may be located in a different second computer system, the second computer system being connected to the first computer system via a network (e.g., the Internet). The second computer system may provide the program instructions to the first computer system for execution. The term "storage medium" may include two or more storage media that may be located in different locations (e.g., different computer systems connected via a network). The storage medium may store program instructions (e.g., embodied as a computer program) executable by one or more processors.

[0110] The model training apparatus, human body posture detection apparatus, storage medium, and computer device according to the above embodiments can execute the methods according to the embodiments of the present application, and have corresponding functional modules and effects by executing the methods. For technical details not described in detail in the above embodiments, reference can be made to the methods according to the embodiments of the present application.

Claims

1. Inputting an image training sample corresponding to a current iteration into a current pose detection network model, and obtaining a first loss function corresponding to the current iteration based on a data format of the image training sample; performing a reprojection process on a current output result of the pose detection network model based on the obtained camera parameters, and obtaining a second loss function corresponding to the current iteration based on a result of the reprojection process; Backpropagating the posture detection network model according to a fitting loss function formed by the first loss function and the second loss function to obtain a posture detection network model for a next iteration, and entering the next iteration until an iteration termination condition is met, thereby realizing training for the posture detection network model; Including, The posture detection network model is a heat map regression network model constructed using a fully convolutional network structure; The data format of the image training sample is a natural scene sample image-keypoint two-dimensional coordinate label format, or a constrained scene sample image-keypoint two-dimensional coordinate and keypoint three-dimensional coordinate label format; Obtaining a first loss function corresponding to the current iteration based on a data format of the image training samples, When the data format of the image training sample is a label format of a constrained scene sample image-keypoint two-dimensional coordinates and keypoint three-dimensional coordinates, obtain a current heatmap set output from the pose detection network model, where the current heatmap set includes a second current probability heatmap and a group of current three-dimensional coordinate heatmaps; Obtain a predetermined standard heatmap set, and obtain a first loss function corresponding to the current iteration based on the standard heatmap set and the current heatmap set; Including, Wherein, the standard heatmap set is determined by transformation of keypoint 3D coordinates in the image training sample, and the standard heatmap set includes a second standard probability heatmap and a group of standard 3D coordinate heatmaps; Model training methods.

2. Obtaining a first loss function corresponding to the current iteration based on a data format of the image training samples, When the data format of the image training sample is a natural scene sample image-keypoint two-dimensional coordinate label format, extracting a first current probability heat map from a current output result of the pose detection network model; Obtain a first standard probability heat map determined in advance, and obtain a first loss function corresponding to the current iteration based on the first standard probability heat map and the first current probability heat map; Including, Wherein the first standard probability heatmap is determined by a transformation to keypoint two-dimensional coordinates in the image training sample. The method of claim 1.

3. Obtaining a first loss function corresponding to the current iteration based on the first standard probability heat map and the first current probability heat map, determining a first difference equation between data corresponding to the first standard probability heat map and data corresponding to the first current probability heat map; determining a first squared equation of a Euclidean norm formed based on the first difference equation as a first loss function corresponding to the current iteration; Including, The method of claim 2.

4. Obtaining a first loss function corresponding to the current iteration based on the standard heatmap set and the current heatmap set, determining a second difference equation between the data corresponding to the second standard probability heat map and the data corresponding to the second current probability heat map; determining a third difference equation between the data corresponding to the standard 3D coordinate heat map group and the data corresponding to the current 3D coordinate heat map group; forming a second squared equation of a Euclidean norm based on the second difference equation, and forming a third squared equation of a Euclidean norm based on the third difference equation; summing the second squared expression and the third squared expression to construct a first loss function corresponding to the current iteration; Including, The method of claim 1.

5. performing a reprojection process on a current output result of the pose detection network model based on the obtained camera parameters; Obtaining camera parameters included in the image training samples or obtaining camera parameters output from one output branch of the pose detection network model corresponding to the image training samples; extracting a current 3D abscissa heat map and a current 3D ordinate heat map from a current output result of the pose detection network model; determining a current 2D abscissa heat map and a current 2D ordinate heat map of keypoints in the image training sample by combining a set reprojection formula according to the camera parameters, the current 3D abscissa heat map and the current 3D ordinate heat map; a current two-dimensional abscissa heat map formed by the current two-dimensional abscissa heat map and the current two-dimensional ordinate heat map is a result of the reprojection process; Including, The method of claim 1.

6. Obtaining a second loss function corresponding to the current iteration based on a result of the reprojection process, Obtaining a set of pre-defined standard 2D coordinate heatmaps, where the set of standard 2D coordinate heatmaps is determined by transformation to keypoint 2D coordinates or keypoint 3D coordinates in the image training samples, and includes a standard 2D abscissa heatmap and a standard 2D ordinate heatmap; determining a fourth difference equation between the data corresponding to the standard two-dimensional coordinate heat map group and the data corresponding to the current two-dimensional abscissa heat map group; determining a fourth squared equation of a Euclidean norm formed based on the fourth difference equation as a second loss function corresponding to the current iteration; Including, The method according to claim 5.

7. The method of claim 1 , wherein the fitting loss function is a sum of the first loss function and the second loss function.

8. The method of claim 1 , wherein the sample images in the image training sample are monocular images captured by a monocular camera.

9. Acquiring an actual image of a person to be detected; inputting the real person scene image into a preset pose detection network model, where the preset pose detection network model is obtained by training with the method according to any one of claims 1 to 8; performing human body posture detection for the person in the actual person scene image based on an output result of the preset posture detection network model, thereby obtaining three-dimensional spatial position coordinates of a plurality of key points of the person; Including, A method for detecting human body posture.

10. the output results include a probability heat map and a set of 3D coordinate heat maps of a plurality of keypoints; performing human body pose detection on the person in the actual person scene image based on an output result of the preset pose detection network model, thereby obtaining three-dimensional spatial position coordinates of a plurality of key points of the person; For each keypoint, determine a probability coordinate point corresponding to a maximum probability value from the probability heat map, and obtain a current position coordinate of the probability coordinate point; searching for a value corresponding to the current position coordinate in each of the three-dimensional abscissa heat map, the three-dimensional ordinate heat map, and the three-dimensional depth coordinate heat map included in the group of three-dimensional coordinate heat maps; constructing three-dimensional spatial position coordinates for each of said key points based on the located values; Including, 10. The method of claim 9.

11. and converting the real person image into a corresponding cartoon person image based on three-dimensional spatial position coordinates of a plurality of key points of the person in the real person image.

11. The method according to claim 9 or 10.

12. a first information determination module configured to input image training samples corresponding to a current iteration into a current pose detection network model, and obtain a first loss function corresponding to the current iteration based on a data format of the image training samples; a second information determination module configured to perform a reprojection process on a current output result of the pose detection network model based on the obtained camera parameters, and obtain a second loss function corresponding to the current iteration based on a result of the reprojection process; a model training module configured to back-propagate the posture detection network model according to a fitting loss function formed by the first loss function and the second loss function to obtain a posture detection network model for a next iteration, and enter the next iteration until an iteration termination condition is met, thereby realizing training for the posture detection network model; Including, The posture detection network model is a heat map regression network model constructed using a fully convolutional network structure; The data format of the image training sample is a natural scene sample image-keypoint two-dimensional coordinate label format, or a constrained scene sample image-keypoint two-dimensional coordinate and keypoint three-dimensional coordinate label format; Obtaining a first loss function corresponding to the current iteration based on a data format of the image training samples, When the data format of the image training sample is a label format of a constrained scene sample image-keypoint two-dimensional coordinates and keypoint three-dimensional coordinates, obtain a current heatmap set output from the pose detection network model, where the current heatmap set includes a second current probability heatmap and a group of current three-dimensional coordinate heatmaps; Obtain a predetermined standard heatmap set, and obtain a first loss function corresponding to the current iteration based on the standard heatmap set and the current heatmap set; Including, Wherein, the standard heatmap set is determined by transformation of keypoint 3D coordinates in the image training sample, and the standard heatmap set includes a second standard probability heatmap and a group of standard 3D coordinate heatmaps; Model training equipment.

13. An information acquisition module configured to acquire a real scene image of a person to be detected; An information input module configured to input the person real scene image to a preset pose detection network model, where the preset pose detection network model is obtained by training according to the method according to any one of claims 1 to 8; an information determination module configured to perform human body pose detection on the person in the real person scene image based on an output result of the preset pose detection network model, thereby obtaining three-dimensional spatial position coordinates of a plurality of key points of the person; Including, Human body posture detection device.

14. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the computing device implements the method of any one of claims 1 to 11.

15. A computer readable storage medium having a computer program stored thereon, the computer program implementing the method of any one of claims 1 to 11 when said program is executed by a processor.

Images