Three-dimensional key point detection model training method and related equipment

Abstract

The invention provides a training method of a three-dimensional (3D) key point detection model, the 3D key point detection model adopts a double-branch structure, the two branches share the same base network, one branch is a 3D position detection network obtained by training based on a second data set, and is mainly used for predicting depth information of key points; and the other branch is a two-dimensional (2D) position detection network obtained by training based on a third data set and is mainly used for predicting 2D position information of key points. Based on the method, the invention further provides a 3D key point detection method, a 3D key point detection device, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to a training method for a three-dimensional key point detection model, a three-dimensional key point detection method, a storage medium, and a program product. Background Art

[0002] In a three-dimensional (3D) human body full-body driving scenario, a 3D key point detection model is usually used to detect the key points of the human body in the input image, and output the 3D position information of the key points of the human body. Then, the 3D position information of the key points of the human body is further used to complete the driving of the virtual objects in the virtual scene. It can be seen that 3D key point detection is one of the basic technologies for 3D human body full-body driving. In other words, the accuracy of the 3D position information of the key points of the human body output by the 3D key point detection model will directly affect the driving effect of the virtual objects. Therefore, how to design a 3D key point detection model to improve the accuracy of the 3D position information of the key points output by the 3D key point detection model is one of the key technical issues that need to be solved in the application of 3D human body full-body driving. Summary of the invention

[0003] In view of this, an embodiment of the present disclosure provides a training method for a 3D key point detection model, wherein the above-mentioned 3D key point detection model includes: a pre-trained base network, a 3D position detection network, a 2D position detection network and a fusion module; wherein the training method includes:

[0004] Acquire a second image and 3D position annotations of key points corresponding to the second image from a second data set; input the second image into the pre-trained base network and the 3D position detection network to obtain a first 3D heat map of each predicted key point; determine a second 3D heat map of each key point in the second image based on the 3D position annotations of the key points corresponding to the second image; determine a loss based on the first 3D heat map and the second 3D heat map, and adjust parameters of the 3D position detection network based on the loss until the 3D position detection network converges; and

[0005] Acquire a third image and 2D position annotations of key points corresponding to the third image from a third data set; input the 2D image into the pre-trained base network and the 2D position detection network to obtain a first 2D heat map of each predicted key point; determine a second 2D heat map of each key point in the third image based on the 2D position annotations of the key points corresponding to the 2D image; determine a loss based on the first 2D heat map and the second 2D heat map, and adjust the parameters of the 2D position detection network based on the loss until the 2D position detection network converges.

[0006] In an embodiment of the present disclosure, the training process of the pre-trained base network includes: obtaining a first image and a human posture annotation corresponding to the first image from a first data set; inputting the first image into a human posture prediction model composed of a base network to obtain a predicted human posture; determining a first loss based on the predicted human posture and the human posture annotation corresponding to the first image, and adjusting the parameters of the human posture prediction model based on the first loss until the human posture prediction model converges; and fixing the parameters of the base network in the human posture prediction model to obtain the pre-trained base network.

[0007] In an embodiment of the present disclosure, the above method may further include: fixing parameters of the 3D position detection network after the 3D position detection network converges.

[0008] In an embodiment of the present disclosure, the above method may further include: before inputting the 2D image into the pre-trained base network and the 2D position detection network, initializing the parameters of the 2D position detection network using the parameters of the 3D position detection network.

[0009] In an embodiment of the present disclosure, the pre-trained base network is a convolutional network, which is used to extract image features of an input image; the 3D position detection network includes: a first convolutional layer and a 3D position determination module; wherein the first convolutional layer is used to output a 3D heat map of each key point in the image based on the image features; and the 3D position determination module is used to determine the 3D position information of each key point based on the 3D heat map of each key point; and the 2D position detection network includes: a second convolutional layer and a 2D position determination module; wherein the second convolutional layer is used to output a 2D heat map of each key point in the image based on the image features; and the 2D position determination module is used to determine the 2D position information of each key point based on the 3D heat map of each key point.

[0010] In an embodiment of the present disclosure, the 3D position detection network further includes: a first deconvolution layer, a second deconvolution layer and a first upsampling layer; the first deconvolution layer, the second deconvolution layer and the first upsampling layer are used to improve the resolution of the input image features; and the 2D position detection network further includes: a third deconvolution layer, a fourth deconvolution layer and a second upsampling layer; the third deconvolution layer, the fourth deconvolution layer and the second upsampling layer are used to improve the resolution of the input image features.

[0011] In an embodiment of the present disclosure, the above method may further include: before inputting the 2D image into the pre-trained base network and the 2D position detection network, using the parameter values ​​of the parameters of the first deconvolution layer, the second deconvolution layer and the first upsampling layer in the 3D position detection network as the initial values ​​of the parameters of the third deconvolution layer, the fourth deconvolution layer and the second upsampling layer in the 2D position detection network.

[0012] In an embodiment of the present disclosure, the first data set is a 2D image data set with human body posture annotations; the second data set is composed of a 3D data part and a 2D data part in a pre-set ratio; wherein the 3D data part is a 3D image data set with key point 3D position annotations; and the 2D data part is a 2D image data set with key point 3D position annotations; and the third data set is a 2D image data set with key point 2D position annotations.

[0013] In an embodiment of the present disclosure, the depth information of the key points in the 3D position annotation of the key points corresponding to the 2D image in the 2D data part is a preset default value.

[0014] In an embodiment of the present disclosure, the preset ratio is 10:1 to 15:1.

[0015] An embodiment of the present disclosure also provides a 3D key point detection method, comprising: extracting image features of an image containing a human body through a base network; detecting 3D position information of each key point of the human body in the image through a 3D position detection network based on the image features; detecting 2D position information of each key point of the human body in the image through a two-dimensional 2D position detection network based on the image features; and fusing the 3D position information and the 2D position information to obtain the 3D position of each key point.

[0016] In an embodiment of the present disclosure, fusing the 3D position information and the 2D position information includes: extracting depth information of each key point from the 3D position information; determining the 2D position information of each key point from the 2D position information; and combining the depth information of each key point with its corresponding 2D position information to obtain the 3D position of each key point.

[0017] In an embodiment of the present disclosure, fusing the 3D position information and the 2D position information includes: extracting depth information and first 2D position information of each key point from the 3D position information; determining second 2D position information of each key point from the 2D position information; processing the first 2D position information and the second 2D position information based on pre-set conditions to obtain 2D position information of each key point; and combining the depth information of each key point with its corresponding 2D position information to obtain the 3D position of each key point.

[0018] In an embodiment of the present disclosure, processing the first 2D position information and the second 2D position information based on a preset condition includes: selecting one from the first 2D position information and the second 2D position information as the 2D position information of the key point based on the position of the key point.

[0019] In an embodiment of the present disclosure, processing the first 2D position information and the second 2D position information based on a preset condition includes: performing calculations on the first 2D position information and the second 2D position information based on a preset calculation rule, and using the calculation results as the 2D position information of the key point.

[0020] The embodiment of the present disclosure further provides a 3D key point detection device, comprising:

[0021] A feature extraction module, for extracting image features of images containing human bodies through a base network;

[0022] A 3D position information determination module, used to detect the 3D position information of each key point of the human body in the image through a 3D position detection network based on the image features;

[0023] a 2D position information determination module, configured to detect 2D position information of each key point of a human body in the image through a 2D position detection network based on the image features; and

[0024] The fusion module is used to fuse the 3D position information and the 2D position information to obtain the 3D position of each key point.

[0025] An embodiment of the present disclosure further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0026] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the above method.

[0027] An embodiment of the present disclosure further provides a computer program product, including computer program instructions. When the computer program instructions are executed on a computer, the computer is enabled to execute the above method.

[0028] From the structure of the above 3D key point detection model, it can be seen that the above 3D key point detection model adopts a dual-branch structure, and the two branches share the same base network, one of which is mainly used to predict the depth information of the key point; the other branch is mainly used to predict the 2D position information of the key point. By predicting the depth information and 2D position information separately and then fusing them, the prediction accuracy of the depth information can be improved while ensuring the accuracy of the 2D position information, thereby ensuring the performance and prediction accuracy of the above 3D key point detection model as a whole.

[0029] In addition, the training method of the 3D key point detection model described in the embodiment of the present disclosure is a multi-stage model training method. During training, the training data set used introduces a large amount of 2D training data with 2D position annotations of key points in open scenes on the basis of image training data with 3D position annotations of key points. Therefore, it is possible to effectively improve the robustness of the 3D key point detection model in open scenes while ensuring that the depth of key points predicted by the 3D key point detection model is not damaged, and effectively solve the problem of high training cost and limited generalization of the 3D key point detection model after training due to the high cost of acquiring image training data with 3D position annotations of key points and the single background, and the low accuracy of the predicted 3D key points in real usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 A schematic diagram showing a 3D human body full body driving scene according to an embodiment of the present disclosure is shown;

[0032] Figure 2 A schematic diagram of the structure of a 3D key point detection model according to some embodiments of the present disclosure is shown;

[0033] Figure 3 The specific structure of the 3D position detection network 220 according to some embodiments of the present disclosure is shown;

[0034] Figure 4The specific structure of the 2D position detection network 230 according to some embodiments of the present disclosure is shown;

[0035] Figure 5 The training process of the 3D key point detection model described in the embodiment of the present disclosure is shown;

[0036] Figure 6 The implementation process of the 3D key point detection method described in the embodiment of the present disclosure is shown;

[0037] Figure 7 The internal structure of the 3D key point detection device according to the embodiment of the present disclosure is shown;

[0038] Figure 8 A more specific schematic diagram of the hardware structure of an electronic device described in some embodiments of the present disclosure. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0040] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0041] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0042] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0043] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0044] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0045] Figure 1 A schematic diagram of a 3D human body full body driving scene according to an embodiment of the present disclosure is shown. Figure 1 In the scene shown, the user can first collect an image containing a human body through the image acquisition device 110. The above image can generally refer to a video containing multiple image frames. Next, the above image acquisition device 110 can transmit the collected image to the 3D key point detection model 120, and the 3D key point detection model 120 processes the received image to obtain the 3D position information of each key point on the human body at each time. It should be noted that, for the convenience of description, the key points of the human body are uniformly referred to as key points in the following text. That is to say, in the embodiment of the present disclosure, the key points generally refer to the key points pre-set on the human body. Those skilled in the art can understand that the above human body key points can include the head key points, neck key points, shoulder key points, arm key points, hand key points, hip key points, leg key points, foot key points, etc. of the human body. The key points of each human body detected will correspond one by one to each key point on the limbs of the virtual image driven by the human body. After the 3D key point detection model 120 obtains the 3D position information of each key point at each moment, the 3D key point detection model 120 outputs the 3D position information of each key point at each moment to the virtual image driving device 130, and the virtual image driving device 120 generates a virtual image that can follow the movement of the human body in the virtual space based on the 3D position information of each key point at each moment.

[0046] As mentioned above, the above-mentioned 3D key point detection of the human body is one of the basic technologies for 3D human body full body driving. The accuracy of the 3D position information of the key points output by the above-mentioned 3D key point detection model 120 will directly affect the driving effect of the virtual object. In order to achieve the goal of accurately outputting the 3D position of the key points, it is usually necessary to use 3D images with 3D position annotations of the key points as training data to train a deep neural network to implement the above-mentioned 3D key point detection model 120. However, the acquisition cost of 3D image training data with 3D position annotations of key points is high, and the background is single. Therefore, it will lead to high cost and limited generalization of the trained 3D key point detection model, and low accuracy of the predicted 3D key points in real usage scenarios.

[0047] In order to solve the above problems, an embodiment of the present disclosure provides a new 3D key point detection model. Figure 2 The structure of the 3D key point detection model described in the embodiment of the present disclosure is shown. Figure 2 As shown, the 3D key point detection model described in the embodiment of the present disclosure may include: a base network 210, a 3D position detection network 220, a two-dimensional (2D) information detection network 230 and a fusion module 240.

[0048] The base network 210 is mainly used to extract features from an input image containing a human body to obtain image features of the image.

[0049] In some embodiments of the present disclosure, the base network 210 may be implemented by a convolutional network. Specifically, the base network 210 may be implemented by a residual network. For example, Resnet may be used to implement the base network 210. In this case, the image features of the image generally refer to the convolutional features of the image.

[0050] In some embodiments of the present disclosure, the size of the image feature output by the base network 210 may be, for example, 1*7*7*2048. It should be noted that the size of the image feature is only an example, and in the embodiments of the present disclosure, the image features of other specifications output by the base network 210 are not limited by the embodiments of the present disclosure.

[0051] The 3D position detection network 220 is mainly used to predict the 3D position information of each key point of the human body in the image according to the image features output by the base network 210.

[0052] In some embodiments of the present disclosure, the above-mentioned 3D position information generally refers to three-dimensional position information including horizontal direction, vertical direction and depth direction. Generally, the above-mentioned 3D position information can also be referred to as UVD position information or UVD coordinates, wherein U represents the horizontal direction; V represents the vertical direction; and D represents the depth direction. In some embodiments of the present disclosure, the above-mentioned UVD coordinates can be specifically divided into two parts: UV coordinates and D coordinates. Among them, the UV coordinates of a certain key point represent the position of the key point in the image coordinate system; the D coordinates of the key point represent the relative depth of the key point relative to the root node. Among them, the above-mentioned root node can be a key point among the above-mentioned key points that are pre-set. For example, the hip key point in the key point can be defined as the root node, and other key points can be used as child nodes of the root node or child nodes of other child nodes. In other words, the key points of the human body can constitute a tree structure. It can be seen that the D coordinates of other key points can be determined based on the D coordinates of the root node.

[0053] In some embodiments of the present disclosure, the 3D position detection network 220 may include a convolution layer and a 3D position determination module. The convolution layer may output a 3D heat map of each key point based on the input image features. The 3D heat map of each key point may also be referred to as a 3D thermal distribution map of each key point, wherein each 3D heat map represents the Gaussian distribution of a key point in a preset space. In some embodiments of the present disclosure, the 3D position determination module may determine the 3D position information of each key point based on the 3D heat map of each key point. For example, the 3D position determination module may obtain the UVD coordinate of a key point by integrating the 3D heat map of a key point with the position information. Specifically, for example, the function argmax or the soft extreme parameter function softargmax may be used to determine the UVD coordinate of a key point based on its 3D heat map.

[0054] Furthermore, in order to improve the accuracy of the UVD coordinates output by the 3D position detection network 220, the 3D position detection network 220 may further include: at least one deconvolution layer and an upsampling layer before the convolution layer, for improving the resolution of the input image features. Then, the image features after the resolution is improved are sequentially input into the convolution layer and the 3D position determination module, so as to further obtain the UVD coordinates of each key point. Since the resolution of the input image features is pre-improved before generating the 3D heat map of the key points, the accuracy of the UVD coordinates output by the 3D position detection network 220 will also be improved accordingly.

[0055] Figure 3 2 shows the specific structure of the 3D position detection network 220 described in some embodiments of the present disclosure. Figure 3As shown, the 3D position detection network 220 may include: a first deconvolution layer 310, a second deconvolution layer 320, a first upsampling layer 330, a first convolution layer 340, and a 3D position determination module 350. Assuming that the input image feature is a convolution feature of size 1*7*7*2048, and the number of key points to be detected is 28, the above feature can increase the resolution of the image feature by 4 times after passing through two deconvolution layers 310, 320 and the first upsampling layer 330, and obtain a high-resolution image feature of 1*28*28*512. Then, the high-resolution image feature can obtain an image feature of 1*28*28*28*28 after passing through the 1*1 first convolution layer 340. Among them, the first 28 represents the number of key points, and the size of the 3D heat map of each key point is 28*28*28. Finally, the UVD coordinates of the above 28 key points are determined respectively by the 3D position determination module 350 based on the 3D heat maps of the above 28 key points.

[0056] The 2D position detection network 230 is mainly used to predict the 2D position information of each key point of the human body in the image according to the image features output by the base network 210. The 2D position information generally refers to two-dimensional position information including horizontal and vertical directions. Generally, the 2D information may also be referred to as UV position information or UV coordinates. It can be seen that the 2D position information does not include information in the depth direction.

[0057] In some embodiments of the present disclosure, the 2D position detection network 230 may include a convolution layer and a 2D position determination module. The convolution layer may output a 2D heat map of each key point based on the input image features. The 2D heat map of each key point may also be referred to as a 2D thermal distribution map of each key point, wherein each 2D heat map represents the Gaussian distribution of a key point in the image coordinate system. In some embodiments of the present disclosure, the 2D position determination module may determine the 2D position information of each key point based on the 2D heat map of each key point. For example, the 2D position determination module may obtain the UV coordinates of a key point by integrating the 2D heat map of a key point with the position information. Specifically, for example, the function argmax or the soft extreme parameter function softargmax may be used to determine the UV coordinates of each key point based on the 2D heat map of each key point.

[0058] Furthermore, in order to improve the accuracy of the UV coordinates output by the 2D position detection network 230, the 2D position detection network 230 may further include: at least one deconvolution layer and an upsampling layer before the convolution layer, for improving the resolution of the input image features. The image features with improved resolution are then sequentially input into the convolution layer and the 2D position determination module to obtain the UV coordinates of each key point. Since the resolution of the input image features is pre-improved before generating the 2D heat map of the key points, the accuracy of the UV coordinates output by the 2D position detection network 230 will also be improved accordingly.

[0059] Figure 4 2 shows the specific structure of the 2D position detection network 230 described in some embodiments of the present disclosure. Figure 4 As shown, the 2D position detection network 230 may include: a third deconvolution layer 410, a fourth deconvolution layer 420, a second upsampling layer 430, a second convolution layer 440, and a 2D position determination module 450. Assuming that the input image feature is a convolution feature of size 1*7*7*2048, and the number of key points to be detected is 28, the above feature can increase the resolution of the image feature by 4 times after passing through two sets of deconvolution layers 410, 420 and the second upsampling layer 430, and obtain a high-resolution image feature of 1*28*28*512. Then, the high-resolution image feature can obtain an image feature of size 1*28*28*28 after passing through the 1*1 second convolution layer 440. Among them, the first 28 represents the number of key points, and the size of the 2D heat map of each key point is 28*28. Finally, the UV coordinates of the above 28 key points are determined respectively by the 2D position determination module 450 based on the 2D heat maps of the above 28 key points.

[0060] From the above Figure 3 and Figure 4 It can be seen that the structures of the above-mentioned 2D position detection network 230 and the above-mentioned 3D position detection network 220 are basically the same. The difference between the two in structure is that the number of channels of the convolution layer (the second convolution layer 440) of the 2D position detection network 230 is different from the number of channels of the convolution layer (the first convolution layer 340) of the 3D position detection network 220. In other words, the output of the convolution layer of the 2D position detection network 230 is a 2D heat map of each key point, which is only used to predict the UV coordinates of each key point. The output of the convolution layer of the 3D position detection network 220 is a 3D heat map of each key point, which is used to predict the UVD coordinates of each key point.

[0061] It should be noted that the above Figure 3 and Figure 4The structure of two deconvolution layers plus one upsampling layer shown in is only an example. The above-mentioned 3D position detection network 220 and 2D position detection network 230 may also include a combination of multiple groups of the above-mentioned deconvolution layers (each group includes one or two deconvolution layers) and upsampling layers to further improve the resolution of the input image features.

[0062] The fusion module 240 is mainly used to fuse the 3D position information of each key point of the human body output by the 3D position detection network 220 and the 2D position information of each key point of the human body output by the 2D position detection network 230 to determine the 3D position of each key point of the human body.

[0063] In some embodiments of the present disclosure, the fusion module 240 may extract the depth information of each key point from the 3D position information of each key point of the human body output by the 3D position detection network 220. The fusion module 240 may also extract the UV position information of each key point from the 2D information of each key point of the human body output by the 2D position detection network 230. Finally, the fusion module 240 combines the extracted depth information of each key point and the UV position information of each key point to obtain the UVD position information of each key point. It should be noted that the operations of extracting the depth information of each key point and the operations of extracting the UV position information of each key point may be performed in parallel or sequentially in any order, and the present disclosure does not limit this.

[0064] In some other embodiments of the present disclosure, the fusion module 240 may first extract the depth information of each key point and the first UV position information of each key point from the 3D position information of each key point of the human body output by the 3D position detection network 220. The fusion module 240 may also extract the second UV position information of each key point from the 2D information of each key point of the human body output by the 2D position detection network 230. Next, the fusion module 240 may also process the first UV position information and the second UV position information according to a pre-set condition to obtain the UV position information of each key point. The pre-set condition may include a processing method for the first 2D position information and the second 2D position information of each key point. For example, the fusion module 240 may select a target UV position information from the first UV position information and the second UV position information as the UV position information of the key point based on the position of the key point. Alternatively, the fusion module 240 may perform a predefined operation on the first UV position information and the second UV position information based on a pre-set operation rule, such as averaging or weighted averaging, and use the calculation result as the UV position information of the key point. Finally, the fusion module 240 combines the extracted depth information of each key point and the UV position information of each key point to obtain the UVD position information of each key point. It should be noted that the operations of extracting the depth information and the first UV position information of each key point and extracting the second UV position information of each key point can be performed in parallel or in any order, and the present disclosure does not limit this.

[0065] From the structure of the above 3D key point detection model, it can be seen that the above 3D key point detection model adopts a dual-branch structure (3D position detection network 220 and 2D position detection network 230), and the two branches share the same base network 210, one of which (3D position detection network 220) is mainly used to predict the depth information of the key point; the other branch (2D position detection network 230) is mainly used to predict the UV position information of the key point. By predicting the depth information and UV position information separately and then fusing them, the prediction accuracy of the depth information can be improved while ensuring the accuracy of the UV position information, thereby ensuring the performance and prediction accuracy of the above 3D key point detection model as a whole.

[0066] The following will further explain in detail the method for training the above 3D key point detection model with reference to the accompanying drawings and examples.

[0067] Before training the model, it is necessary to prepare a training data set. In the embodiment of the present disclosure, different training data sets need to be prepared for the base network 210, the 3D position detection network 220, and the 2D position detection network 230 in the above 3D key point detection model.

[0068] In some embodiments of the present disclosure, the base network 210 may be trained using a first data set. In some embodiments of the present disclosure, a 2D image data set with human body posture annotations may be used as the first data set. Specifically, the first data set may be selected from a public data set.

[0069] In some embodiments of the present disclosure, the 3D position detection network 220 is trained using a second data set. In some embodiments of the present disclosure, the second data set may be composed of a 3D data part and a 2D data part according to a preset ratio. The 3D data part may be composed of 3D image data with key point 3D position annotations. The 2D data part may be generated using 2D image data with key point 2D position annotations. The 3D data part may generally be motion capture data consisting of a three-dimensional RGB image and key point 3D position annotations obtained by a multi-eye RGB motion capture device. The 2D data part may be obtained by manually annotating a two-dimensional RGB image. In addition, the preset ratio may generally be a ratio of 10:1 to 15:1, so that the proportion of the 3D data part in the second data set reaches more than 90% or even more. Further, considering that the scene, environment, lighting and other conditions of the 3D data shooting are relatively simple, in order to improve the overall generalization ability of the 3D key point detection model, when selecting the data set, the images in the 2D data part are preferably different in scene, environment, lighting and other aspects. In addition, since the annotation information corresponding to the 2D image is usually the 2D position information of the key points, and does not include the depth information of the key points, it is inconsistent with the form of the 3D position information of the key points annotated in the 3D data part. Therefore, the acquired 2D image and its corresponding 2D position annotation of the key points cannot be directly added to the second data set, but it is necessary to first generate the 3D position information of the key points based on the 2D position information of the key points annotated by each 2D image, and add it as the 3D position annotation of the key points corresponding to the 2D image to generate the above-mentioned 2D data part. Specifically, the 3D position annotation of the key points can be uniformly generated by adding depth information to the 2D position annotation of the key points corresponding to the 2D image, so that the annotation information of the 2D data part is consistent with the annotation information of the 3D data part in form, which is convenient for model training. Among them, in some embodiments of the present disclosure, the added depth information can be a preset default value, for example, 0.

[0070] In some embodiments of the present disclosure, the 2D position detection network 230 is trained using a third data set. In an embodiment of the present disclosure, the third data set may be composed of 2D image data with key point 2D position annotations. Similarly, the third data set may also be obtained by manually annotating two-dimensional RGB images. Similarly, in order to improve the overall generalization ability of the 3D key point detection model, when selecting the data set, the images in the third data set are preferably different in scenes, environments, lighting, etc.

[0071] After the first data set, the second data set and the third data set are prepared, the training of the base network, the 3D position detection network and the 2D position detection network can be completed in sequence respectively.

[0072] Figure 5 The training process of the 3D key point detection model described in the embodiment of the present disclosure is shown. Figure 5 As shown, the above method may include the following steps:

[0073] In step 502, a first image and a human body posture annotation corresponding to the first image are obtained from a first data set.

[0074] As mentioned above, in the embodiment of the present disclosure, the first image is a 2D image.

[0075] In step 504, the first image is input into a human body posture prediction model composed of the base network to obtain a predicted human body posture.

[0076] As mentioned above, the base network is used to extract the image features of the input image. Therefore, when training the base network, another network that can predict human posture based on image features can be added to it for joint training, thereby completing the training of the base network. In the embodiment of the present disclosure, the network that predicts human posture based on image features is called a posture prediction network. That is, the human posture prediction model is composed of the base network and the posture prediction network. Among them, the base network is still used to extract the image features of the input image; the posture prediction network is used to predict human posture based on the extracted image features. The training task set for the human posture prediction model composed of the base network and the posture prediction network can be a task of predicting human posture. In the embodiment of the present disclosure, the training of the human posture prediction model can be completed in a supervised manner based on the first data set, and the trained base network is used as the base network used in the embodiment of the present disclosure.

[0077] In step 506, a first loss is determined according to the predicted human body posture and the human body posture annotation corresponding to the first image;

[0078] In step 508, the parameters of the human posture prediction model composed of the base network are adjusted based on the first loss.

[0079] Repeat the above steps 502 to 508 until the above human posture prediction model converges. After the above human posture prediction model converges, the base network therein can be retained as the base network used in this application. In other words, the training of the base network can be completed by cyclically executing the above steps 502 to 508. In the embodiments of the present disclosure, the trained base network can also be referred to as a pre-trained base network.

[0080] After the base network is trained, the parameters of the base network can be fixed, and then subsequent steps are performed to complete the training of the 3D position detection network.

[0081] In step 512, a second image and 3D position annotations of key points corresponding to the second image are obtained from a second data set.

[0082] It should be noted that, as mentioned above, the second image in the second data set includes both 3D images and 2D images. The ratio of 3D images to 2D images in the second data set is a preset ratio, such as 10:1. Furthermore, for the 2D images in the second data set, the depth information in the 3D position annotation of the key points corresponding thereto may be a preset default value, such as 0.

[0083] In step 514, the second image is input into the pre-trained base network and 3D position detection network to obtain a first 3D heat map of each predicted key point.

[0084] In step 516, a second 3D heat map of each key point in the second image is determined based on the 3D position annotations of the key points corresponding to the second image.

[0085] At step 518, a second loss is determined according to the first 3D heat map and the second 3D heat map;

[0086] In step 520 , parameters of the 3D position detection network are adjusted based on the second loss.

[0087] The above steps 512 to 520 are repeatedly performed until the above 3D position detection network converges. After the above 3D position detection network converges, it indicates that the training of the 3D position detection network has been completed.

[0088] After the training of the 3D position detection network is completed, the parameters of the 3D position detection network can be further fixed, and then the training of the 2D position detection network can be continued.

[0089] In step 524, a third image and 2D position annotations of key points corresponding to the third image are obtained from the third data set.

[0090] As mentioned above, in the embodiment of the present disclosure, the third image is also a 2D image.

[0091] In step 526, the third image is input into the pre-trained base network and the 2D position detection network to obtain a first 2D heat map of each predicted key point.

[0092] In step 528, a second 2D heat map of each key point in the third image is determined based on the 2D position annotations of the key points corresponding to the third image.

[0093] In step 530 , a third loss is determined according to the first 2D heat map and the second 2D heat map.

[0094] In step 532, parameters of the 2D position detection network are adjusted based on the third loss.

[0095] Repeat steps 524 to 532 until the 3D position detection network converges. After the 2D position detection network converges, it indicates that the training of the 2D position detection network has been completed. At this point, the training of the entire 3D key point detection model is completed.

[0096] It should be noted that in order to further improve the training speed of the 3D key point detection model, improve the training efficiency, and enhance the accuracy of the 3D key point detection model, in some embodiments of the present disclosure, before executing the above step 524, the following step 522 may be further executed.

[0097] In step 522, the parameters of the 2D position detection network are initialized using the parameters of the 3D position detection network.

[0098] Specifically, in the embodiment of the present disclosure, the structures of the above 3D position detection network and the above 2D position detection network are basically the same except for the number of channels of the convolution layer (the first convolution layer and the second convolution layer). Therefore, except for the convolution layer, the parameters of the above 2D position detection network can be directly initialized to the parameters determined after the training of the above 3D position detection network. For the convolution layer (the second convolution layer), random initialization can be performed to obtain the initial values ​​of the convolution layer parameters.

[0099] The above method makes full use of the structural similarities between the 3D position detection network and the 2D position detection network, and directly initializes the parameters of the 2D position detection network based on the parameters of the trained 3D position detection network, which can achieve the effect of rapid convergence of the model during training, thereby improving the training efficiency of the 2D position detection network. Furthermore, since the third training set is different from the second training set, more samples with differences in scenes, environments, lighting, etc. are used for training, so that the generalization ability of the entire model can be improved on the basis of efficient training, thereby improving the overall accuracy of the 3D key point detection model.

[0100] In the above-mentioned multi-stage model training method, a large amount of 2D training data in open scenes is introduced on the basis of the image training data with key point 3D position annotations. Therefore, the robustness of the 3D key point detection model in open scenes can be effectively improved while ensuring that the depth of key points predicted by the 3D key point detection model is not damaged. This effectively solves the problem of limited generalization of the trained 3D key point detection model due to the high cost of acquiring image training data with key point 3D position annotations and the single background, and the low accuracy of the predicted 3D key points in real usage scenarios.

[0101] Based on the above 3D key point detection model, an embodiment of the present disclosure also provides a 3D key point detection method.

[0102] Figure 6 The implementation process of the 3D key point detection method described in the embodiment of the present disclosure is shown. Figure 6 As shown, the above 3D key point detection method may include the following steps:

[0103] In step 602, image features of an image containing a human body are extracted through a base network;

[0104] In step 604, based on the image features, 3D position information of each key point of the human body in the image is detected by a 3D position detection network;

[0105] In step 606, based on the image features, 2D position information of each key point of the human body in the image is detected by a 2D position detection network; and

[0106] In step 608, the 3D position information and the 2D position information are fused to obtain the 3D position of each key point.

[0107] It should be noted that the above steps 604 and 606 may be executed in parallel or in series in any order. The embodiment of the present disclosure does not limit the execution order of the above steps 604 and 606.

[0108] In addition, in the embodiment of the present disclosure, the above step 602 can be implemented by the base network trained by the above 3D key point detection model training method. The above step 604 can be implemented by the 3D position detection network trained by the above 3D key point detection model training method. And the above step 606 can be implemented by the 2D position detection network trained by the above 3D key point detection model training method. It will not be described in detail here.

[0109] For the above 608, the above 3D position information and the above 2D position information may be fused through various methods.

[0110] Specifically, in some embodiments of the present disclosure, the above step 608 may include:

[0111] First, extract the depth information of each key point from the above 3D position information and determine the 2D position information of each key point from the above 2D position information;

[0112] Then, the depth information of each key point is combined with its corresponding 2D position information to obtain the 3D position of each key point.

[0113] In some other embodiments of the present disclosure, the above step 608 may include:

[0114] First, extract the depth information and the first 2D position information of each key point from the above 3D position information and determine the second 2D position information of each key point from the above 2D position information;

[0115] Then, the first 2D position information and the second 2D position information are processed based on a preset condition to obtain 2D position information of each key point;

[0116] Finally, the depth information of each key point is combined with its corresponding 2D position information to obtain the 3D position of each key point.

[0117] Among them, in some embodiments of the present disclosure, the above-mentioned pre-set conditions may include a method for processing the first 2D position information and the second 2D position information of each key point. For example, one may be selected from the first 2D position information and the second 2D position information based on the position of the key point as the 2D position information of the key point; or the first 2D position information and the second 2D position information may be operated (for example, averaged or weighted averaged) based on a pre-set operation rule, and the operation result may be used as the 2D position information of the key point, etc.

[0118] Based on the above 3D key point detection method, an embodiment of the present disclosure further provides a 3D key point detection device.

[0119] Figure 7 The internal structure of the 3D key point detection device described in the embodiment of the present disclosure is shown. Figure 7 As shown, the above 3D key point detection device may include the following modules:

[0120] A feature extraction module 702, for extracting image features of an image containing a human body through a base network;

[0121] A 3D position information determination module 704, configured to detect 3D position information of each key point of a human body in the image through a 3D position detection network based on the image features;

[0122] A 2D position information determination module 706, configured to detect 2D position information of each key point of a human body in the image through a 2D position detection network based on the image features; and

[0123] The fusion module 708 is used to fuse the 3D position information and the 2D position information to obtain the 3D position of each key point.

[0124] The specific implementation of each of the above modules can refer to the aforementioned methods and drawings. The training methods of the models or networks used by each of the above modules can also refer to the aforementioned methods and drawings, which will not be repeated here.

[0125] For the convenience of description, the above device is described in various modules according to their functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware. The device of the above embodiment is used to implement the training method of the corresponding three-dimensional key point detection model in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0126] In the above-mentioned 3D key point detection method and device, the 3D key point detection model used adopts a dual-branch structure, and the two branches share the same base network, one of which is mainly used to predict the depth information of the key point; the other branch is mainly used to predict the 2D position information of the key point. By predicting the depth information and the 2D position information separately and then fusing them, the prediction accuracy of the depth information can be improved while ensuring the accuracy of the 2D position information, thereby ensuring the performance and prediction accuracy of the above-mentioned 3D key point detection model as a whole. In addition, when training the above-mentioned model, the training data set used introduces a large amount of 2D training data with 2D position annotations of key points in open scenes on the basis of the image training data with 3D position annotations of key points. Therefore, while ensuring that the depth of the key points predicted by the 3D key point detection model is not damaged, the robustness of the 3D key point detection model in open scenes can be effectively improved, and the generalization of the 3D key point detection model after training is limited due to the high cost of obtaining the image training data with 3D position annotations of key points and the single background, and the problem of low accuracy of the 3D key points predicted in real usage scenarios.

[0127] Figure 8 A more specific hardware structure diagram of an electronic device provided in this embodiment, the device may include: a processor 2010, a memory 2020, an input / output interface 2030, a communication interface 2040, and a bus 2050. The processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040 are connected to each other in communication within the device through the bus 2050.

[0128] The processor 2010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0129] The memory 2020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 2020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 2020 and are called and executed by the processor 2010.

[0130] The input / output interface 2030 is used to connect input / output devices to realize information input and output. The input / output devices can be configured in the device as components, or can be externally connected to the device to provide corresponding functions. The input devices can include microphones, various sensors, etc., and the output devices can include displays, speakers, vibrators, indicator lights, etc.

[0131] The communication interface 2040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0132] The bus 2050 includes a path that transmits information between the various components of the device (eg, the processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040).

[0133] It should be noted that, although the above device only shows the processor 2010, the memory 2020, the input / output interface 2030, the communication interface 2040 and the bus 2050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0134] The electronic device of the above-mentioned embodiment is used to implement the corresponding 3D key point detection model training method or 3D key point detection method in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0135] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the 3D key point detection model training method or the 3D key point detection method as described in any of the above embodiments.

[0136] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0137] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the 3D key point detection model training method or 3D key point detection method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0138] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0139] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0140] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0141] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A training method for a three-dimensional 3D key point detection model, the 3D key point detection model comprising: A pre-trained base network, a 3D position detection network, a two-dimensional 2D position detection network and a fusion module; wherein the training method of the 3D key point detection model includes: Acquire a second image and 3D position annotations of key points corresponding to the second image from a second data set; input the second image into the pre-trained base network and the 3D position detection network to obtain a first 3D heat map of each predicted key point; determine a second 3D heat map of each key point in the second image based on the 3D position annotations of the key points corresponding to the second image; determine a second loss based on the first 3D heat map and the second 3D heat map, and adjust the parameters of the 3D position detection network based on the second loss until the 3D position detection network converges; and A third image and 2D position annotations of key points corresponding to the third image are obtained from a third data set; the 2D image is input into the pre-trained base network and the 2D position detection network to obtain a first 2D heat map of each predicted key point; a second 2D heat map of each key point in the third image is determined based on the 2D position annotations of the key points corresponding to the 2D image; a third loss is determined based on the first 2D heat map and the second 2D heat map, and parameters of the 2D position detection network are adjusted based on the third loss until the 2D position detection network converges.

2. The method according to claim 1, wherein: The training method of the pre-trained base network includes: Acquire a first image and a human body posture annotation corresponding to the first image from a first data set; Inputting the first image into a human posture prediction model composed of a base network to obtain a predicted human posture; Determining a first loss according to the predicted human posture and the human posture annotation corresponding to the first image, and adjusting parameters of the human posture prediction model based on the first loss until the human posture prediction model converges; and The parameters of the base network in the human posture prediction model are fixed to obtain the pre-trained base network.

3. The method according to claim 1, further comprising: After the 3D position detection network converges, the parameters of the 3D position detection network are fixed.

4. The method according to claim 1, further comprising: Before inputting the 2D image into the pre-trained base network and the 2D position detection network, the parameters of the 2D position detection network are initialized using the parameters of the 3D position detection network.

5. The method according to claim 1, wherein: The pre-trained base network is a convolutional network, which is used to extract image features of the input image; The 3D position detection network comprises: a first convolutional layer and a 3D position determination module; wherein the first convolutional layer is used to output a 3D heat map of each key point in the image based on the image features; and the 3D position determination module is used to determine the 3D position information of each key point based on the 3D heat map of each key point; and The 2D position detection network includes: a second convolutional layer and a 2D position determination module; wherein the second convolutional layer is used to output a 2D heat map of each key point in the image based on the image features; and the 2D position determination module is used to determine the 2D position information of each key point based on the 3D heat map of each key point.

6. The method according to claim 5, wherein: The 3D position detection network further includes: a first deconvolution layer, a second deconvolution layer, and a first upsampling layer; the first deconvolution layer, the second deconvolution layer, and the first upsampling layer are used to improve the resolution of the image features; and The 2D position detection network further includes: a third deconvolution layer, a fourth deconvolution layer and a second upsampling layer; the third deconvolution layer, the fourth deconvolution layer and the second upsampling layer are used to improve the resolution of the image features.

7. The method according to claim 6, further comprising: Before the 2D image is input into the pre-trained base network and the 2D position detection network, the parameter values ​​of the parameters of the first deconvolution layer, the second deconvolution layer and the first upsampling layer in the 3D position detection network are respectively used as the initial values ​​of the corresponding parameters of the third deconvolution layer, the fourth deconvolution layer and the second upsampling layer in the 2D position detection network.

8. The method according to claim 2, wherein: The first data set is a 2D image data set with human body posture annotations; The second data set is composed of a 3D data part and a 2D data part according to a preset ratio; wherein the 3D data part is a 3D image data set with key point 3D position annotations; and the 2D data part is a 2D image data set with key point 3D position annotations; and The third data set is a 2D image data set with key point 2D position annotations.

9. The method according to claim 8, wherein: The depth information of the key points in the 3D position annotation of the key points corresponding to the 2D image in the 2D data part is a preset default value.

10. The method according to claim 8, wherein: The preset ratio is 10:1 to 15:

1.

11. A three-dimensional 3D key point detection method, comprising: Extracting image features of images containing human bodies through a base network; Detecting 3D position information of each key point of a human body in the image through a 3D position detection network based on the image features; Based on the image features, detecting 2D position information of each key point of the human body in the image through a two-dimensional 2D position detection network; as well as The 3D position information and the 2D position information are fused to obtain the 3D position of each key point.

12. The method according to claim 11, wherein: The fusing of the 3D position information and the 2D position information includes: Extracting depth information of each key point from the 3D position information; Determining 2D position information of each key point from the 2D position information; and The depth information of each key point is combined with its corresponding 2D position information to obtain the 3D position of each key point.

13. The method according to claim 11, wherein: The fusing of the 3D position information and the 2D position information includes: Extracting depth information and first 2D position information of each key point from the 3D position information; Determine second 2D position information of each key point from the 2D position information; Processing the first 2D position information and the second 2D position information based on a preset condition to obtain 2D position information of each key point; and The depth information of each key point is combined with its corresponding 2D position information to obtain the 3D position of each key point.

14. The method according to claim 13, wherein: Processing the first 2D position information and the second 2D position information based on a preset condition includes: Based on the position of the key point, one of the first 2D position information and the second 2D position information is selected as the 2D position information of the key point.

15. The method according to claim 13, wherein: Processing the first 2D position information and the second 2D position information based on a preset condition includes: The first 2D position information and the second 2D position information are calculated based on a preset calculation rule, and the calculation result is used as the 2D position information of the key point.

16. A 3D key point detection device, comprising: A feature extraction module, for extracting image features of images containing human bodies through a base network; A 3D position information determination module, used to detect the 3D position information of each key point of the human body in the image through a 3D position detection network based on the image features; A two-dimensional 2D position information determination module, used to detect the 2D position information of each key point of the human body in the image through a 2D position detection network based on the image features; as well as The fusion module is used to fuse the 3D position information and the 2D position information to obtain the 3D position of each key point.

17. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 15 is implemented.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method of any one of claims 1 to 15.

19. A computer program product, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 15.

Images