Method for training three-dimensional key point detection model and related device

Abstract

The present disclosure provides a method for training a three-dimensional (3D) key point detection model. The 3D key point detection model uses a dual-branch structure, and two branches share a same backbone network, wherein one of the branches is a 3D position detection network trained on the basis of a second data set and is mainly used for predicting depth information of a key point, and the other of the branches is a two-dimensional (2D) position detection network trained on the basis of a third data set and is mainly used for predicting 2D position information of the key point. On the basis of said method, the present disclosure further provides a 3D key point detection method, a 3D key point detection apparatus, a storage medium, and a program product.

Description

Training method and related equipment for 3D key point detection model

[0001] This application claims priority to the Chinese invention patent application entitled “Training method and related equipment for three-dimensional key point detection model” and application number 202311279536.7, filed on September 28, 2023. The entire contents of that application are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of Internet technology, and in particular to a training method for a three-dimensional key point detection model, a three-dimensional key point detection method, a storage medium, and a program product. Background Art

[0003] In a three-dimensional (3D) human body full-body driving scenario, a 3D key point detection model is usually used to detect the key points of the human body in the input image and output the 3D position information of the key points of the human body. Then, the 3D position information of the key points of the human body is further used to complete the driving of the virtual objects in the virtual scene. It can be seen that 3D key point detection is one of the basic technologies for 3D human body full-body driving. That is to say, the accuracy of the 3D position information of the key points of the human body output by the 3D key point detection model will directly affect the driving effect of the virtual objects. Therefore, how to design a 3D key point detection model to improve the accuracy of the 3D position information of the key points output by the 3D key point detection model is one of the key technical issues that need to be solved in the application of 3D human body full-body driving.

[0004] Summary of the Invention

[0005] In view of this, an embodiment of the present disclosure provides a training method for a 3D key point detection model, wherein the above-mentioned 3D key point detection model includes: a pre-trained base network, a 3D position detection network, a 2D position detection network and a fusion module; wherein the training method includes: obtaining a second image and 3D position annotations of key points corresponding to the second image from a second data set; inputting the second image into the pre-trained base network and the 3D position detection network to obtain a first 3D heat map of each key point predicted; determining a second 3D heat map of each key point in the second image based on the 3D position annotations of the key points corresponding to the second image; and determining a second 3D heat map of each key point in the second image based on the first 3D heat map and the second 3D heat map. D heat map determines the loss, and adjusts the parameters of the 3D position detection network based on the loss until the 3D position detection network converges; and obtains a third image and the 2D position annotations of the key points corresponding to the third image from a third data set; inputs the 2D image into the pre-trained base network and the 2D position detection network to obtain a first 2D heat map of each predicted key point; determines a second 2D heat map of each key point in the third image based on the 2D position annotations of the key points corresponding to the 2D image; determines the loss according to the first 2D heat map and the second 2D heat map, and adjusts the parameters of the 2D position detection network based on the loss until the 2D position detection network converges.

[0006] In an embodiment of the present disclosure, the training process of the pre-trained base network includes: obtaining a first image and a human posture annotation corresponding to the first image from a first data set; inputting the first image into a human posture prediction model composed of a base network to obtain a predicted human posture; determining a first loss based on the predicted human posture and the human posture annotation corresponding to the first image, and adjusting the parameters of the human posture prediction model based on the first loss until the human posture prediction model converges; and fixing the parameters of the base network in the human posture prediction model to obtain the pre-trained base network.

[0007] In an embodiment of the present disclosure, the above method may further include: fixing parameters of the 3D position detection network after the 3D position detection network converges.

[0008] In an embodiment of the present disclosure, the above method may further include: before inputting the 2D image into the pre-trained base network and the 2D position detection network, initializing the parameters of the 2D position detection network using the parameters of the 3D position detection network.

[0009] In an embodiment of the present disclosure, the pre-trained base network is a convolutional network, which is used to extract image features of the input image; the 3D position detection network includes: a first convolutional layer and a 3D position determination module; wherein the first convolutional layer is used to output a 3D heat map of each key point in the image based on the image features; and the 3D position determination module is used to determine the 3D position information of each key point based on the 3D heat map of each key point; and the 2D position detection network includes: a second convolutional layer and a 2D position determination module; wherein the second convolutional layer is used to output a 2D heat map of each key point in the image based on the image features; and the 2D position determination module is used to determine the 2D position information of each key point based on the 3D heat map of each key point.

[0010] In an embodiment of the present disclosure, the 3D position detection network further includes: a first deconvolution layer, a second deconvolution layer, and a first upsampling layer; the first deconvolution layer, the second deconvolution layer, and the first upsampling layer are used to improve the resolution of the input image features; and the 2D position detection network further includes: a third deconvolution layer, a fourth deconvolution layer, and a second upsampling layer; the third deconvolution layer, the fourth deconvolution layer, and the second upsampling layer are used to improve the resolution of the input image features.

[0011] In an embodiment of the present disclosure, the above method may further include: before inputting the 2D image into the pre-trained base network and the 2D position detection network, using the parameter values ​​of the parameters of the first deconvolution layer, the second deconvolution layer, and the first upsampling layer in the 3D position detection network as the initial values ​​of the parameters of the third deconvolution layer, the fourth deconvolution layer, and the second upsampling layer in the 2D position detection network.

[0012] In an embodiment of the present disclosure, the first data set is a 2D image data set with human body posture annotations; the second data set is composed of a 3D data portion and a 2D data portion combined in a pre-set ratio; wherein the 3D data portion is a 3D image data set with key point 3D position annotations; and the 2D data portion is a 2D image data set with key point 3D position annotations; and the third data set is a 2D image data set with key point 2D position annotations.

[0013] In an embodiment of the present disclosure, the depth information of the key points in the 3D position annotations of the key points corresponding to the 2D image in the 2D data portion is a preset default value.

[0014] In an embodiment of the present disclosure, the preset ratio is 10:1 to 15:1.

[0015] An embodiment of the present disclosure also provides a 3D key point detection method, including: extracting image features of an image containing a human body through a base network; detecting 3D position information of each key point of the human body in the image through a 3D position detection network based on the image features; detecting 2D position information of each key point of the human body in the image through a two-dimensional 2D position detection network based on the image features; and fusing the 3D position information and the 2D position information to obtain the 3D position of each key point.

[0016] In an embodiment of the present disclosure, fusing the 3D position information and the 2D position information includes: extracting the depth information of each key point from the 3D position information; determining the 2D position information of each key point from the 2D position information; and combining the depth information of each key point with its corresponding 2D position information to obtain the 3D position of each key point.

[0017] In an embodiment of the present disclosure, fusing the 3D position information and the 2D position information includes: extracting depth information and first 2D position information of each key point from the 3D position information; determining second 2D position information of each key point from the 2D position information; processing the first 2D position information and the second 2D position information based on pre-set conditions to obtain 2D position information of each key point; and combining the depth information of each key point with its corresponding 2D position information to obtain the 3D position of each key point.

[0018] In an embodiment of the present disclosure, processing the first 2D position information and the second 2D position information based on a preset condition includes: selecting one from the first 2D position information and the second 2D position information as the 2D position information of the key point based on the position of the key point.

[0019] In an embodiment of the present disclosure, processing the first 2D position information and the second 2D position information based on a preset condition includes: performing calculations on the first 2D position information and the second 2D position information based on a preset calculation rule, and using the calculation results as the 2D position information of the key point.

[0020] An embodiment of the present disclosure also provides a 3D key point detection device, including: a feature extraction module, used to extract image features of an image containing a human body through a base network; a 3D position information determination module, used to detect the 3D position information of each key point of the human body in the image through a 3D position detection network based on the image features; a 2D position information determination module, used to detect the 2D position information of each key point of the human body in the image through a 2D position detection network based on the image features; and a fusion module, used to fuse the 3D position information and the 2D position information to obtain the 3D position of each key point.

[0021] An embodiment of the present disclosure further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0022] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the above method.

[0023] An embodiment of the present disclosure further provides a computer program product, comprising computer program instructions, which enable the computer to execute the above method when the computer program instructions are executed on a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] FIG1 shows a schematic diagram of a 3D human body full-body driving scene according to an embodiment of the present disclosure;

[0026] FIG2 shows a schematic diagram of the structure of a 3D key point detection model according to some embodiments of the present disclosure;

[0027] FIG3 shows a specific structure of the 3D position detection network 220 according to some embodiments of the present disclosure;

[0028] FIG4 shows a specific structure of a 2D position detection network 230 according to some embodiments of the present disclosure;

[0029] FIG5 shows the training process of the 3D key point detection model according to an embodiment of the present disclosure;

[0030] FIG6 shows the implementation process of the 3D key point detection method according to an embodiment of the present disclosure;

[0031] FIG7 shows the internal structure of the 3D key point detection device according to an embodiment of the present disclosure;

[0032] FIG8 is a schematic diagram of a more specific hardware structure of an electronic device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0034] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0035] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0036] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the disclosed technical solution based on the prompt message.

[0037] As an optional but non-limiting implementation, in response to a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0038] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0039] The structure of the aforementioned 3D keypoint detection model demonstrates its dual-branch architecture, with both branches sharing the same base network. One branch is primarily responsible for predicting the depth information of keypoints, while the other branch is primarily responsible for predicting their 2D position information. This separate prediction and subsequent fusion of depth and 2D position information improves the accuracy of depth prediction while maintaining the accuracy of 2D position information, thereby ensuring the overall performance and prediction accuracy of the 3D keypoint detection model.

[0040] In addition, the training method of the 3D key point detection model described in the embodiment of the present disclosure is a multi-stage model training method. During training, the training data set used not only introduces a large amount of 2D training data with 2D position annotations of key points in open scenes on the basis of image training data with 3D position annotations of key points, but also introduces a large amount of 2D training data with 2D position annotations of key points in open scenes. Therefore, while ensuring that the depth of key points predicted by the 3D key point detection model is not damaged, the robustness of the 3D key point detection model in open scenes can be effectively improved, and the problem of high training cost and limited generalization of the 3D key point detection model after training, which is caused by the high cost of acquiring image training data with 3D position annotations and a single background, and the low accuracy of the predicted 3D key points in real usage scenarios, can be effectively solved.

[0041] Figure 1 shows a schematic diagram of a 3D human body full-body driving scenario according to an embodiment of the present disclosure. In the scenario shown in Figure 1, a user can first capture an image containing a human body using an image acquisition device 110. This image can typically be a video containing multiple image frames. Next, the image acquisition device 110 can transmit the captured image to a 3D keypoint detection model 120, which processes the received image to obtain 3D position information for each keypoint on the human body at each moment. It should be noted that for ease of description, the keypoints of the human body will be referred to as "keypoints" in the following text. That is, in the embodiments of the present disclosure, the keypoints generally refer to pre-defined keypoints on the human body. Those skilled in the art will understand that the keypoints of the human body can include keypoints on the head, neck, shoulders, arms, hands, hips, legs, feet, and so on. Each detected keypoint on the human body will correspond one-to-one with each keypoint on the limbs of the virtual avatar driven by the human body. After the above-mentioned 3D key point detection model 120 obtains the 3D position information of each key point at each moment, the above-mentioned 3D key point detection model 120 outputs the 3D position information of each key point at each moment to the virtual image driving device 130, and the virtual image driving device 120 generates a virtual image in the virtual space that can follow the movement of the human body based on the 3D position information of each key point at each moment.

[0042] As mentioned above, the above-mentioned 3D key point detection of the human body is one of the basic technologies for 3D human body full-body driving. The accuracy of the 3D position information of the key points output by the above-mentioned 3D key point detection model 120 will directly affect the driving effect of the virtual object. In order to achieve the goal of accurately outputting the 3D positions of the key points, it is usually necessary to use 3D images with 3D position annotations of the key points as training data to train a deep neural network to implement the above-mentioned 3D key point detection model 120. However, the acquisition cost of 3D image training data with 3D position annotations of the key points is high, and the background is single. Therefore, it will lead to high cost and limited generalization of the trained 3D key point detection model, and low accuracy of the predicted 3D key points in real usage scenarios.

[0043] To address the above issues, embodiments of the present disclosure provide a new 3D keypoint detection model. Figure 2 shows the structure of the 3D keypoint detection model described in embodiments of the present disclosure. As shown in Figure 2, the 3D keypoint detection model described in embodiments of the present disclosure may include: a base network 210, a 3D position detection network 220, a two-dimensional (2D) information detection network 230, and a fusion module 240.

[0044] The base network 210 mainly performs feature extraction on an input image containing a human body to obtain image features of the image.

[0045] In some embodiments of the present disclosure, the base network 210 may be implemented using a convolutional network. Specifically, the base network 210 may be implemented using a residual network. For example, the base network 210 may be implemented using ResNet. In this case, the image features of the image generally refer to the convolutional features of the image.

[0046] In some embodiments of the present disclosure, the size of the image feature output by the base network 210 may be, for example, 1*7*7*2048. It should be noted that the size of the image feature is only an example, and in the embodiments of the present disclosure, the base network 210 may output image features of other sizes, which are not limited by the embodiments of the present disclosure.

[0047] The 3D position detection network 220 is mainly used to predict the 3D position information of each key point of the human body in the image according to the image features output by the base network 210.

[0048] In some embodiments of the present disclosure, the above-mentioned 3D position information generally refers to three-dimensional position information including horizontal, vertical and depth directions. Generally, the above-mentioned 3D position information can also be referred to as UVD position information or UVD coordinates, wherein U represents the horizontal direction; V represents the vertical direction; and D represents the depth direction. In some embodiments of the present disclosure, the above-mentioned UVD coordinates can be specifically divided into two parts: UV coordinates and D coordinates. Among them, the UV coordinates of a certain key point represent the position of the key point in the image coordinate system; the D coordinates of the key point represent the relative depth of the key point relative to the root node. Among them, the above-mentioned root node can be a key point among the above-mentioned key points that are pre-set. For example, the hip key point among the key points can be defined as the root node, and other key points can be used as child nodes of the root node or child nodes of other child nodes. In other words, the key points of the human body can form a tree structure. It can be seen that the D coordinates of other key points can be determined based on the D coordinates of the root node.

[0049] In some embodiments of the present disclosure, the 3D position detection network 220 may include a convolutional layer and a 3D position determination module. The convolutional layer may output a 3D heat map of each key point based on the input image features. The 3D heat map of each key point may also be referred to as a 3D thermal distribution map of each key point, wherein each 3D heat map represents the Gaussian distribution of a key point in a preset space. In some embodiments of the present disclosure, the 3D position determination module may determine the 3D position information of each key point based on the 3D heat map of each key point. For example, the 3D position determination module may obtain the UVD coordinates of a key point by integrating the 3D heat map of a key point with the position information. Specifically, for example, the function argmax or the soft-extreme parameter function softargmax may be used to determine the UVD coordinates of a key point based on its 3D heat map.

[0050] Furthermore, to improve the accuracy of the UVD coordinates output by the 3D position detection network 220, the 3D position detection network 220 may further include at least one deconvolution layer and one upsampling layer before the convolution layer to increase the resolution of the input image features. The image features with increased resolution are then sequentially input into the convolution layer and the 3D position determination module to further determine the UVD coordinates of each key point. Because the resolution of the input image features is pre-enhanced before generating the 3D heat map of the key points, the accuracy of the UVD coordinates output by the 3D position detection network 220 is also correspondingly improved.

[0051] Figure 3 shows the specific structure of the 3D position detection network 220 described in some embodiments of the present disclosure. As shown in Figure 3, the 3D position detection network 220 may include a first deconvolution layer 310, a second deconvolution layer 320, a first upsampling layer 330, a first convolution layer 340, and a 3D position determination module 350. Assuming the input image features are convolutional features of size 1*7*7*2048, and the number of keypoints to be detected is 28, then after passing through the two deconvolution layers 310 and 320 and the first upsampling layer 330, the resolution of the image features can be increased by 4 times, resulting in high-resolution image features of size 1*28*28*512. Subsequently, the high-resolution image features pass through the 1*1 first convolution layer 340 to obtain image features of size 1*28*28*28*28. The first 28 represents the number of keypoints, and the size of the 3D heat map for each keypoint is 28*28*28. Finally, the UVD coordinates of the 28 key points are determined based on the 3D heat maps of the 28 key points by the 3D position determination module 350 .

[0052] The 2D position detection network 230 is primarily used to predict the 2D position information of key points of the human body in the image based on the image features output by the base network 210. The 2D position information generally refers to two-dimensional position information including horizontal and vertical directions. This 2D information is also commonly referred to as UV position information or UV coordinates. This 2D position information does not include depth information.

[0053] In some embodiments of the present disclosure, the above-mentioned 2D position detection network 230 may include a convolution layer and a 2D position determination module. The convolution layer may output a 2D heat map of each key point based on the input image features. The 2D heat map of each key point may also be referred to as a 2D thermal distribution map of each key point, wherein each 2D heat map represents the Gaussian distribution of a key point in the image coordinate system. In some embodiments of the present disclosure, the above-mentioned 2D position determination module may respectively determine the 2D position information of each key point based on the 2D heat map of each key point. For example, the above-mentioned 2D position determination module may obtain the UV coordinates of a key point by integrating the 2D heat map of a key point with the position information. Specifically, for example, the function argmax or the soft extreme parameter function softargmax may be used to determine the UV coordinates of each key point based on the 2D heat map of each key point.

[0054] Furthermore, to improve the accuracy of the UV coordinates output by the 2D position detection network 230, the 2D position detection network 230 may further include at least one deconvolution layer and one upsampling layer before the convolution layer to increase the resolution of the input image features. The image features with increased resolution are then sequentially input into the convolution layer and the 2D position determination module to obtain the UV coordinates of each key point. Because the resolution of the input image features is pre-enhanced before generating the 2D heat map of the key points, the accuracy of the UV coordinates output by the 2D position detection network 230 is also correspondingly improved.

[0055] Figure 4 illustrates the specific structure of the 2D position detection network 230 described in some embodiments of the present disclosure. As shown in Figure 4 , the 2D position detection network 230 may include a third deconvolution layer 410, a fourth deconvolution layer 420, a second upsampling layer 430, a second convolution layer 440, and a 2D position determination module 450. Assuming the input image features are convolutional features of size 1*7*7*2048, and the number of keypoints to be detected is 28, then these features, after passing through two sets of deconvolution layers 410 and 420 and the second upsampling layer 430, can increase the resolution of the image features by a factor of four, resulting in high-resolution image features of size 1*28*28*512. These high-resolution image features then pass through a second 1*1 convolution layer 440, resulting in image features of size 1*28*28*28. The first 28 represents the number of keypoints, and the size of the 2D heatmap for each keypoint is 28*28. Finally, the UV coordinates of the 28 key points are determined respectively based on the 2D heat maps of the 28 key points by the 2D position determination module 450 .

[0056] As can be seen from Figures 3 and 4 above, the structures of the above-mentioned 2D position detection network 230 and the above-mentioned 3D position detection network 220 are basically the same. The difference in structure between the two is that the number of channels of the convolution layer (second convolution layer 440) of the 2D position detection network 230 is different from the number of channels of the convolution layer (first convolution layer 340) of the 3D position detection network 220. In other words, the output of the convolution layer of the 2D position detection network 230 is a 2D heat map of each key point, which is only used to predict the UV coordinates of each key point. The output of the convolution layer of the 3D position detection network 220 is a 3D heat map of each key point, which is used to predict the UVD coordinates of each key point.

[0057] It should be noted that the structure of two deconvolution layers plus one upsampling layer shown in Figures 3 and 4 is only an example. The above-mentioned 3D position detection network 220 and the 2D position detection network 230 can also include a combination of multiple groups of the above-mentioned deconvolution layers (each group includes one or two deconvolution layers) and upsampling layers to further improve the resolution of the input image features.

[0058] The fusion module 240 is mainly used to fuse the 3D position information of each key point of the human body output by the 3D position detection network 220 and the 2D position information of each key point of the human body output by the 2D position detection network 230 to determine the 3D position of each key point of the human body.

[0059] In some embodiments of the present disclosure, the fusion module 240 may extract depth information of each key point from the 3D position information of each key point of the human body output by the 3D position detection network 220. The fusion module 240 may also extract UV position information of each key point from the 2D information of each key point of the human body output by the 2D position detection network 230. Finally, the fusion module 240 combines the extracted depth information of each key point and the UV position information of each key point to obtain UVD position information of each key point. It should be noted that the operations of extracting the depth information of each key point and extracting the UV position information of each key point may be performed in parallel or in any order, and the present disclosure does not limit this.

[0060] In other embodiments of the present disclosure, the fusion module 240 may first extract depth information and first UV position information of each key point from the 3D position information of each key point on the human body output by the 3D position detection network 220. The fusion module 240 may also extract second UV position information of each key point from the 2D information of each key point on the human body output by the 2D position detection network 230. Next, the fusion module 240 may process the first UV position information and the second UV position information according to pre-set conditions to obtain UV position information for each key point. The pre-set conditions may include a method for processing the first 2D position information and the second 2D position information of each key point. For example, the fusion module 240 may select a target UV position information from the first UV position information and the second UV position information as the UV position information of the key point based on the position of the key point. Alternatively, the fusion module 240 may perform a predefined operation, such as averaging or weighted averaging, on the first UV position information and the second UV position information based on a pre-set operation rule, and use the result as the UV position information of the key point. Finally, the fusion module 240 combines the extracted depth information and UV position information of each key point to obtain UVD position information for each key point. It should be noted that the operations of extracting the depth information and first UV position information for each key point, as well as extracting the second UV position information for each key point, can be performed in parallel or sequentially in any order, and this is not limited in this disclosure.

[0061] As can be seen from the structure of the above-mentioned 3D key point detection model, the above-mentioned 3D key point detection model adopts a dual-branch structure (3D position detection network 220 and 2D position detection network 230). The two branches share the same base network 210, one of which (3D position detection network 220) is mainly used to predict the depth information of the key points; the other branch (2D position detection network 230) is mainly used to predict the UV position information of the key points. By predicting the depth information and UV position information separately and then fusing them, the prediction accuracy of the depth information can be improved while ensuring the accuracy of the UV position information, thereby ensuring the overall performance and prediction accuracy of the above-mentioned 3D key point detection model.

[0062] The following will further explain in detail the method for training the above 3D key point detection model with reference to the accompanying drawings and examples.

[0063] Before training the model, it is necessary to prepare a training dataset. In the embodiment of the present disclosure, different training datasets are required for the base network 210, the 3D position detection network 220, and the 2D position detection network 230 in the above-mentioned 3D key point detection model.

[0064] In some embodiments of the present disclosure, the base network 210 may be trained using a first dataset. In some embodiments of the present disclosure, a 2D image dataset with human pose annotations may be used as the first dataset. Specifically, the first dataset may be selected from a publicly available dataset.

[0065] In some embodiments of the present disclosure, the 3D position detection network 220 is trained using a second dataset. In some embodiments of the present disclosure, the second dataset may be composed of a 3D data portion and a 2D data portion in a pre-set ratio. The 3D data portion may be composed of 3D image data with keypoint 3D position annotations. The 2D data portion may be generated using 2D image data with keypoint 2D position annotations. The 3D data portion may typically be motion capture data collected by a multi-camera RGB motion capture device, consisting of 3D RGB images and keypoint 3D position annotations. The 2D data portion may be obtained by manually annotating 2D RGB images. Furthermore, the pre-set ratio may typically be 10:1 to 15:1, ensuring that the 3D data portion accounts for 90% or more of the second dataset. Furthermore, given that the scene, environment, lighting, and other conditions used in 3D data capture are relatively uniform, to improve the overall generalization capability of the 3D keypoint detection model, it is preferable that the images in the 2D data portion have varying scenes, environments, and lighting when selecting the dataset. In addition, since the annotation information corresponding to the 2D image is usually the 2D position information of the key points, it does not include the depth information of the key points, and is inconsistent with the form of the 3D position information of the key points annotated in the 3D data part. Therefore, the acquired 2D image and its corresponding 2D position annotation of the key points cannot be directly added to the second data set, but it is necessary to first generate the 3D position information of the key points based on the 2D position information of the key points annotated by each 2D image, and add it as the 3D position annotation of the key points corresponding to the 2D image to generate the above-mentioned 2D data part. Specifically, the depth information can be uniformly added to the 2D position annotation of the key points corresponding to the 2D image to generate the 3D position annotation of the key points, so that the annotation information of the 2D data part is consistent in form with the annotation information of the 3D data part, which is convenient for model training. In some embodiments of the present disclosure, the added depth information can be a preset default value, for example, 0.

[0066] In some embodiments of the present disclosure, the 2D position detection network 230 is trained using a third data set. In embodiments of the present disclosure, the third data set may be composed of 2D image data with key point 2D position annotations. Similarly, the third data set may be obtained by manually annotating two-dimensional RGB images. Similarly, in order to improve the overall generalization capability of the 3D key point detection model, when selecting the data set, the images in the third data set preferably differ in terms of scene, environment, lighting, etc.

[0067] After the first data set, the second data set, and the third data set are prepared, the training of the base network, the 3D position detection network, and the 2D position detection network can be completed in sequence respectively.

[0068] FIG5 shows the training process of the 3D key point detection model according to an embodiment of the present disclosure. As shown in FIG5 , the above method may include the following steps:

[0069] In step 502, a first image and a human body posture annotation corresponding to the first image are obtained from a first data set.

[0070] As mentioned above, in the embodiment of the present disclosure, the first image is a 2D image.

[0071] In step 504, the first image is input into a human body posture prediction model composed of the base network to obtain a predicted human body posture.

[0072] As mentioned above, the above-mentioned base network is used to extract the image features of the input image. Therefore, when training the above-mentioned base network, another network that can predict human posture based on image features can be added to it for joint training, thereby completing the training of the above-mentioned base network. In the embodiment of the present disclosure, the above-mentioned network that predicts human posture based on image features is called a posture prediction network. That is, the above-mentioned human posture prediction model is composed of the above-mentioned base network and the posture prediction network. Among them, the above-mentioned base network is still used to extract the image features of the input image; and the above-mentioned posture prediction network is used to predict human posture based on the extracted image features. The training task set for the human posture prediction model composed of the above-mentioned base network and the posture prediction network can be a task of predicting human posture. In the embodiment of the present disclosure, the training of the human posture prediction model can be completed in a supervised manner based on the above-mentioned first data set, and the trained base network can be used as the base network used in the embodiment of the present disclosure.

[0073] At step 506, a first loss is determined based on the predicted human posture and the human posture annotation corresponding to the first image;

[0074] In step 508, the parameters of the human posture prediction model composed of the base network are adjusted based on the first loss.

[0075] Repeat steps 502 to 508 until the human body posture prediction model converges. After the human body posture prediction model converges, the base network therein can be retained as the base network used in this application. In other words, the base network can be trained by repeatedly executing steps 502 to 508. In the embodiments of the present disclosure, the trained base network can also be referred to as a pre-trained base network.

[0076] After the base network is trained, the parameters of the base network can be fixed, and then subsequent steps can be performed to complete the training of the 3D position detection network.

[0077] In step 512 , a second image and 3D position annotations of key points corresponding to the second image are obtained from a second data set.

[0078] It should be noted that, as previously described, the second images in the second dataset include both 3D and 2D images. The ratio of 3D and 2D images in the second dataset is a pre-set ratio, for example, 10:1. Furthermore, for the 2D images in the second dataset, the depth information in the 3D position annotations of the corresponding key points may be a pre-set default value, for example, 0.

[0079] In step 514, the second image is input into the pre-trained base network and 3D position detection network to obtain a first 3D heat map of each key point predicted.

[0080] In step 516 , a second 3D heat map of each key point in the second image is determined based on the 3D position annotations of the key points corresponding to the second image.

[0081] At step 518 , a second loss is determined based on the first 3D heat map and the second 3D heat map;

[0082] In step 520 , parameters of the 3D position detection network are adjusted based on the second loss.

[0083] Repeat steps 512 to 520 until the 3D position detection network converges. After the 3D position detection network converges, it indicates that the training of the 3D position detection network has been completed.

[0084] After the 3D position detection network is trained, the parameters of the 3D position detection network may be further fixed, and then the 2D position detection network may be trained.

[0085] In step 524 , a third image and 2D position annotations of key points corresponding to the third image are obtained from the third data set.

[0086] As mentioned above, in the embodiment of the present disclosure, the third image is also a 2D image.

[0087] In step 526, the third image is input into the pre-trained base network and 2D position detection network to obtain a first 2D heat map of each key point predicted.

[0088] In step 528 , a second 2D heat map of each key point in the third image is determined based on the 2D position annotations of the key points corresponding to the third image.

[0089] In step 530 , a third loss is determined based on the first 2D heat map and the second 2D heat map.

[0090] In step 532 , parameters of the 2D position detection network are adjusted based on the third loss.

[0091] Repeat steps 524 to 532 until the 3D position detection network converges. After the 2D position detection network converges, the training of the 2D position detection network is complete. At this point, the entire 3D key point detection model training is complete.

[0092] It should be noted that in order to further improve the training speed of the 3D key point detection model, improve the training efficiency, and enhance the accuracy of the 3D key point detection model, in some embodiments of the present disclosure, before executing the above step 524, the following step 522 may be further executed.

[0093] In step 522, the parameters of the 2D position detection network are initialized using the parameters of the 3D position detection network.

[0094] Specifically, in the embodiments of the present disclosure, the structures of the 3D position detection network and the 2D position detection network are substantially identical, except for the number of channels in the convolutional layers (the first convolutional layer and the second convolutional layer). Therefore, except for the convolutional layers, the parameters of the 2D position detection network can be directly initialized to the parameters determined after training the 3D position detection network. The convolutional layer (the second convolutional layer) can be randomly initialized to obtain the initial values ​​of the convolutional layer parameters.

[0095] This method leverages the structural similarities between the 3D and 2D position detection networks, directly initializing the parameters of the 2D position detection network based on the parameters of the trained 3D position detection network. This allows for rapid model convergence during training, thereby improving the training efficiency of the 2D position detection network. Furthermore, because the third training set differs from the second training set, training utilizes more samples with varying scenes, environments, and lighting. This allows for improved generalization of the entire model based on efficient training, thereby enhancing the overall accuracy of the 3D keypoint detection model.

[0096] In the above-mentioned multi-stage model training method, a large amount of 2D training data in open scenes is introduced on the basis of the image training data with key point 3D position annotations. Therefore, the robustness of the 3D key point detection model in open scenes can be effectively improved while ensuring that the depth of the key points predicted by the 3D key point detection model is not damaged. This effectively solves the problem that the generalization of the trained 3D key point detection model is limited due to the high cost of obtaining image training data with key point 3D position annotations and the single background, and the low accuracy of the predicted 3D key points in real usage scenarios.

[0097] Based on the above 3D key point detection model, an embodiment of the present disclosure further provides a 3D key point detection method.

[0098] FIG6 shows the implementation process of the 3D key point detection method according to an embodiment of the present disclosure. As shown in FIG6 , the 3D key point detection method may include the following steps:

[0099] In step 602, image features of an image containing a human body are extracted through a base network;

[0100] In step 604, 3D position information of each key point of the human body in the image is detected by a 3D position detection network based on the image features.

[0101] At step 606, 2D position information of each key point of the human body in the image is detected by a 2D position detection network based on the image features; and

[0102] In step 608 , the 3D position information and the 2D position information are fused to obtain the 3D position of each key point.

[0103] It should be noted that the above steps 604 and 606 may be executed in parallel or in series in any order. The embodiment of the present disclosure does not limit the execution order of the above steps 604 and 606.

[0104] Furthermore, in an embodiment of the present disclosure, step 602 can be implemented using a base network trained using the aforementioned 3D keypoint detection model training method. Step 604 can be implemented using a 3D position detection network trained using the aforementioned 3D keypoint detection model training method. Step 606 can be implemented using a 2D position detection network trained using the aforementioned 3D keypoint detection model training method. This will not be described in detail here.

[0105] Regarding the above 608 , the above 3D position information and the above 2D position information may be fused through various methods.

[0106] Specifically, in some embodiments of the present disclosure, step 608 may include:

[0107] First, extract the depth information of each key point from the above 3D position information and determine the 2D position information of each key point from the above 2D position information;

[0108] Then, the depth information of each key point is combined with its corresponding 2D position information to obtain the 3D position of each key point.

[0109] In some other embodiments of the present disclosure, step 608 may include:

[0110] First, extracting depth information and first 2D position information of each key point from the 3D position information and determining second 2D position information of each key point from the 2D position information;

[0111] Then, the first 2D position information and the second 2D position information are processed based on a preset condition to obtain 2D position information of each key point;

[0112] Finally, the depth information of each key point is combined with its corresponding 2D position information to obtain the 3D position of each key point.

[0113] In some embodiments of the present disclosure, the aforementioned pre-set conditions may include a method for processing the first 2D position information and the second 2D position information of each key point. For example, one of the first 2D position information and the second 2D position information may be selected as the 2D position information of the key point based on the position of the key point; or the first 2D position information and the second 2D position information may be calculated (e.g., averaged or weighted averaged) based on a pre-set calculation rule, and the result of the calculation may be used as the 2D position information of the key point.

[0114] Based on the above 3D key point detection method, an embodiment of the present disclosure further provides a 3D key point detection device.

[0115] FIG7 shows the internal structure of the 3D key point detection device according to an embodiment of the present disclosure. As shown in FIG7 , the 3D key point detection device may include the following modules:

[0116] A feature extraction module 702 is configured to extract image features of an image containing a human body through a base network;

[0117] a 3D position information determination module 704, configured to detect 3D position information of key points of a human body in the image through a 3D position detection network based on the image features;

[0118] a 2D position information determining module 706 , configured to detect 2D position information of key points of a human body in the image through a 2D position detection network based on the image features; and

[0119] The fusion module 708 is configured to fuse the 3D position information and the 2D position information to obtain the 3D position of each key point.

[0120] The specific implementation of each of the above modules can refer to the aforementioned methods and drawings. The training methods of the models or networks used by each of the above modules can also refer to the aforementioned methods and drawings, which will not be repeated here.

[0121] For ease of description, the above devices are described separately by function into various modules. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware. The devices of the above embodiments are used to implement the corresponding 3D keypoint detection model training method of any of the aforementioned embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0122] In the above-mentioned 3D keypoint detection method and device, the 3D keypoint detection model used adopts a dual-branch structure, with the two branches sharing the same base network. One branch is mainly used to predict the depth information of the keypoints; the other branch is mainly used to predict the 2D position information of the keypoints. By predicting the depth information and 2D position information separately and then fusing them, the prediction accuracy of the depth information can be improved while ensuring the accuracy of the 2D position information, thereby ensuring the overall performance and prediction accuracy of the above-mentioned 3D keypoint detection model. In addition, when training the above-mentioned model, the training dataset used also introduces a large amount of 2D training data with 2D position annotations of keypoints in open scenes on the basis of image training data with 3D position annotations of keypoints. Therefore, the robustness of the 3D keypoint detection model in open scenes can be effectively improved while ensuring that the depth of the keypoints predicted by the 3D keypoint detection model is not damaged. This effectively solves the problem that the generalization of the trained 3D keypoint detection model is limited due to the high cost of obtaining image training data with 3D position annotations and the single background, and the low accuracy of the predicted 3D keypoints in real-world usage scenarios.

[0123] FIG8 is a schematic diagram of the hardware structure of a more specific electronic device provided in this embodiment. The device may include: a processor 2010, a memory 2020, an input / output interface 2030, a communication interface 2040, and a bus 2050. The processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040 are communicatively connected to each other within the device via the bus 2050.

[0124] The processor 2010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0125] The memory 2020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 2020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 220 and is called and executed by the processor 2010.

[0126] The input / output interface 2030 is used to connect input / output devices to enable information input and output. Input / output devices can be configured as components within the device or externally connected to the device to provide corresponding functions. Input devices may include microphones and various sensors, while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0127] The communication interface 2040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0128] The bus 2050 comprises a path for transmitting information between the various components of the device (eg, the processor 2010 , the memory 2020 , the input / output interface 2030 , and the communication interface 2040 ).

[0129] It should be noted that although the above device only shows the processor 2010, the memory 2020, the input / output interface 2030, the communication interface 2040, and the bus 2050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0130] The electronic device of the above embodiment is used to implement the training method of the corresponding 3D key point detection model or the 3D key point detection method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0131] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the 3D key point detection model training method or 3D key point detection method as described in any of the above embodiments.

[0132] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0133] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the training method of the 3D key point detection model or the 3D key point detection method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0134] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0135] In addition, to simplify the description and discussion, and so as not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0136] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0137] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A training method for a three-dimensional 3D key point detection model, the 3D key point detection model comprising: A pre-trained base network, a 3D position detection network, a two-dimensional 2D position detection network and a fusion module; wherein the training method of the 3D key point detection model includes: Acquire a second image and 3D position annotations of key points corresponding to the second image from a second data set; input the second image into the pre-trained base network and the 3D position detection network to obtain a first 3D heat map of each predicted key point; determine a second 3D heat map of each key point in the second image based on the 3D position annotations of the key points corresponding to the second image; determine a second loss based on the first 3D heat map and the second 3D heat map, and adjust the parameters of the 3D position detection network based on the second loss until the 3D position detection network converges; and A third image and 2D position annotations of key points corresponding to the third image are obtained from a third data set; the 2D image is input into the pre-trained base network and the 2D position detection network to obtain a first 2D heat map of each predicted key point; a second 2D heat map of each key point in the third image is determined based on the 2D position annotations of the key points corresponding to the 2D image; a third loss is determined based on the first 2D heat map and the second 2D heat map, and parameters of the 2D position detection network are adjusted based on the third loss until the 2D position detection network converges.

2. The method according to claim 1, wherein: The training method of the pre-trained base network includes: Acquire a first image and a human body posture annotation corresponding to the first image from a first data set; Inputting the first image into a human posture prediction model composed of a base network to obtain a predicted human posture; Determining a first loss according to the predicted human posture and the human posture annotation corresponding to the first image, and adjusting parameters of the human posture prediction model based on the first loss until the human posture prediction model converges; and The parameters of the base network in the human posture prediction model are fixed to obtain the pre-trained base network.

3. The method according to claim 1, further comprising: After the 3D position detection network converges, the parameters of the 3D position detection network are fixed.

4. The method according to claim 1, further comprising: Before inputting the 2D image into the pre-trained base network and the 2D position detection network, the parameters of the 2D position detection network are initialized using the parameters of the 3D position detection network.

5. The method according to claim 1, wherein: The pre-trained base network is a convolutional network, which is used to extract image features of the input image; The 3D position detection network comprises: a first convolutional layer and a 3D position determination module; wherein the first convolutional layer is used to output a 3D heat map of each key point in the image based on the image features; and the 3D position determination module is used to determine the 3D position information of each key point based on the 3D heat map of each key point; and The 2D position detection network includes: a second convolutional layer and a 2D position determination module; wherein the second convolutional layer is used to output a 2D heat map of each key point in the image based on the image features; and the 2D position determination module is used to determine the 2D position information of each key point based on the 3D heat map of each key point.

6. The method according to claim 5, wherein: The 3D position detection network further includes: a first deconvolution layer, a second deconvolution layer, and a first upsampling layer; the first deconvolution layer, the second deconvolution layer, and the first upsampling layer are used to improve the resolution of the image features; and The 2D position detection network further includes: a third deconvolution layer, a fourth deconvolution layer and a second upsampling layer; the third deconvolution layer, the fourth deconvolution layer and the second upsampling layer are used to improve the resolution of the image features.

7. The method according to claim 6, further comprising: Before the 2D image is input into the pre-trained base network and the 2D position detection network, the parameter values ​​of the parameters of the first deconvolution layer, the second deconvolution layer and the first upsampling layer in the 3D position detection network are respectively used as the initial values ​​of the corresponding parameters of the third deconvolution layer, the fourth deconvolution layer and the second upsampling layer in the 2D position detection network.

8. The method according to claim 2, wherein: The first data set is a 2D image data set with human body posture annotations; The second data set is composed of a 3D data part and a 2D data part according to a preset ratio; wherein the 3D data part is a 3D image data set with key point 3D position annotations; and the 2D data part is a 2D image data set with key point 3D position annotations; and The third data set is a 2D image data set with key point 2D position annotations.

9. The method according to claim 8, wherein: The depth information of the key points in the 3D position annotation of the key points corresponding to the 2D image in the 2D data part is a preset default value.

10. The method according to claim 8, wherein: The preset ratio is 10:1 to 15:

1.

11. A three-dimensional 3D key point detection method, comprising: Extracting image features of images containing human bodies through a base network; Detecting 3D position information of each key point of a human body in the image through a 3D position detection network based on the image features; Based on the image features, detecting 2D position information of each key point of the human body in the image through a two-dimensional 2D position detection network; as well as The 3D position information and the 2D position information are fused to obtain the 3D position of each key point.

12. The method according to claim 11, wherein: The fusing of the 3D position information and the 2D position information includes: Extracting depth information of each key point from the 3D position information; Determine 2D position information of each key point from the 2D position information; and The depth information of each key point is combined with its corresponding 2D position information to obtain the 3D position of each key point.

13. The method according to claim 11, wherein: The fusing of the 3D position information and the 2D position information includes: Extracting depth information and first 2D position information of each key point from the 3D position information; Determine second 2D position information of each key point from the 2D position information; Processing the first 2D position information and the second 2D position information based on a preset condition to obtain 2D position information of each key point; and The depth information of each key point is combined with its corresponding 2D position information to obtain the 3D position of each key point.

14. The method according to claim 13, wherein: Processing the first 2D position information and the second 2D position information based on a preset condition includes: Based on the position of the key point, one of the first 2D position information and the second 2D position information is selected as the 2D position information of the key point.

15. The method according to claim 13, wherein: Processing the first 2D position information and the second 2D position information based on a preset condition includes: The first 2D position information and the second 2D position information are calculated based on a preset calculation rule, and the calculation result is used as the 2D position information of the key point.

16. A three-dimensional 3D key point detection device, comprising: A feature extraction module, for extracting image features of images containing human bodies through a base network; A 3D position information determination module, used to detect the 3D position information of each key point of the human body in the image through a 3D position detection network based on the image features; A two-dimensional 2D position information determination module, used to detect the 2D position information of each key point of the human body in the image through a 2D position detection network based on the image features; as well as The fusion module is used to fuse the 3D position information and the 2D position information to obtain the 3D position of each key point.

17. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 15 is implemented.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method of any one of claims 1 to 15.

19. A computer program product, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 15.

Images