Motion capture method and apparatus, electronic device, and medium

Abstract

The present disclosure provides a motion capture method, device, equipment and medium, and relates to the technical field of computers. The motion capture method comprises: acquiring a human body region image from a to-be-identified image; inputting the human body region image into a limb capture network to acquire a hand bounding box and a limb motion; acquiring a hand motion according to the hand bounding box; and fusing the hand motion and the limb motion to obtain a human body motion. The present disclosure improves the detection rate of the hand bounding box.

Description

Technical Field

[0001] This disclosure relates to the field of computers, and more particularly to a motion capture method, motion capture device, electronic device, and computer-readable storage medium. Background Technology

[0002] Motion capture methods have wide applications in live entertainment, human-computer interaction, and virtual content creation, such as using motion capture to drive virtual characters in live broadcasts. Specifically, to capture hand movements, it is necessary to obtain a bounding box for the hand, crop the hand region image based on the bounding box, and then identify hand movements from the hand region image.

[0003] In related technologies, a separate hand detection network, such as CenterNet or YOLO, is used to obtain the hand bounding box from the image to be recognized. However, when using a separate hand detection network to obtain the hand bounding box, the entire image needs to be recognized, which involves a large amount of computation. In addition, since the hand region occupies a small proportion of the image to be recognized, the detection rate is low when using a separate hand detection network to obtain the hand bounding box. Furthermore, running the hand detection network requires a lot of computing resources, which leads to a decrease in the frame rate of motion capture. Summary of the Invention

[0004] This disclosure provides a motion capture method, a motion capture device, an electronic device, and a computer-readable storage medium to at least solve the problems of high computational load and low detection rate in related technologies. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a motion capture method is provided, comprising:

[0006] Obtain human body region images from the image to be identified;

[0007] The human body region image is input into a limb capture network to obtain hand bounding boxes and limb movements;

[0008] The hand movement is obtained based on the hand bounding box;

[0009] The hand movements and limb movements are combined to obtain human body movements.

[0010] Optionally, the limb capture network includes a backbone network, a deconvolution layer connected to the backbone network, a first convolution layer connected to the deconvolution layer, and a hand detection branch network connected to the deconvolution layer.

[0011] The step of inputting the human body region image into the limb capture network to obtain the hand bounding box includes:

[0012] The human body region image is input into the backbone network to extract deep features from the human body region image;

[0013] The deep features are input into the deconvolution layer to obtain high-resolution features of the human body region image from the deep features;

[0014] The high-resolution features are input into the first convolutional layer to obtain a two-dimensional key point heatmap of the human body region image;

[0015] The two-dimensional key point heatmap and the high-resolution features are input into the hand detection branch network to obtain the hand bounding box heatmap set of the human body region image;

[0016] The hand bounding frame is determined based on the hand bounding frame heatmap.

[0017] Optionally, the limb capture network further includes a fully connected layer connected to the backbone network;

[0018] The step of inputting the human body region image into the limb capture network to obtain limb movements includes:

[0019] The deep features are input into the fully connected layer to obtain the skeletal rotation parameters of the human body region image;

[0020] The limb movement is determined based on the bone rotation parameters and the two-dimensional key point heatmap.

[0021] Optionally, the hand detection branch network includes a second convolutional layer connected to the deconvolutional layer, a splicing unit connected to the second convolutional layer, and a third convolutional layer connected to the splicing unit;

[0022] The step of inputting the two-dimensional keypoint heatmap and the high-resolution features into the hand detection branch network to obtain a heatmap set of hand bounding boxes in the human body region image includes:

[0023] The high-resolution features are input into the second convolutional layer to obtain the hand bounding box features of the human body region image;

[0024] The hand bounding box feature and the high-resolution feature are input into the stitching unit, and the hand bounding box feature and the high-resolution feature are stitched together to obtain the stitching result;

[0025] The stitching result is input into the third convolutional layer to obtain the hand bounding box heatmap.

[0026] Optionally, determining the hand bounding box based on the hand bounding box heatmap includes:

[0027] The first confidence level of the wrist key points is obtained from the two-dimensional key point heat map;

[0028] The presence of a wrist is determined based on the first confidence level, and the bounding box of the hand is determined based on the determination result.

[0029] Optionally, the hand bounding box heatmap atlas includes a hand bounding box heatmap, a hand bounding box width and height heatmap, and a hand bounding box offset heatmap.

[0030] Determining the hand bounding box based on the judgment result includes:

[0031] When it is determined that both the left and right wrists exist or neither exists, obtain the first maximum value of the probability of the hand bounding box existing in the heatmap of the hand bounding box, the first position coordinates corresponding to the first maximum value, and the second confidence level.

[0032] If the second confidence level is greater than or equal to the preset confidence threshold, then based on the first position coordinates, the first width value, the first height value, and the first offset value of the corresponding hand bounding box are obtained from the hand bounding box width and height heat map and the hand bounding box offset heat map, respectively.

[0033] The hand bounding box is determined based on the first width value, the first height value, and the first offset value.

[0034] Optionally, the hand bounding box heatmap atlas includes a hand bounding box heatmap, a hand bounding box width and height heatmap, and a hand bounding box offset heatmap.

[0035] Determining the hand bounding box based on the judgment result includes:

[0036] When it is determined that only the left wrist or the right wrist exists, extract the bounding box information of the hand corresponding to the existing wrist;

[0037] Based on the hand bounding box information, a suppression operation is performed on the heatmap of the target hand bounding box corresponding to the non-existent wrist, and the hand bounding box is determined based on the suppression operation result.

[0038] Optionally, the hand bounding box information includes the center position coordinates of the hand bounding box, a second width value, and a second height value;

[0039] The step of suppressing the heatmap of the target hand bounding box corresponding to the non-existent wrist based on the hand bounding box information, and determining the hand bounding box based on the suppression operation result, includes:

[0040] Obtain the minimum value between the second height value and the second width value, and determine the target area in the target hand bounding box heat map based on the minimum value and the center position coordinates;

[0041] The probability values ​​corresponding to each location coordinate within the target area are weighted to obtain the suppression operation result;

[0042] Obtain the second maximum value of the probability of a hand bounding box existing in the suppression operation result, the second position coordinates corresponding to the second maximum value, and the fourth confidence level;

[0043] If the fourth confidence level is greater than or equal to the preset confidence threshold, then based on the second position coordinates, the third width value, the third height value, and the second offset value of the corresponding hand bounding box are obtained from the hand bounding box width and height heat map and the hand bounding box offset heat map, respectively.

[0044] The bounding box of the hand corresponding to the non-existent wrist is determined based on the third width value, the third height value, and the second offset value.

[0045] Optionally, before inputting the human body region image into the limb capture network, the method further includes:

[0046] Obtain a training set and train the limb capture network to be trained based on the training set to obtain the limb capture network;

[0047] The training set includes human body region image samples and labeled hand bounding boxes corresponding to the human body region image samples.

[0048] Optionally, training the limb capture network to be trained based on the training set to obtain the limb capture network includes:

[0049] The human body region image samples are input into the limb capture network to be trained to obtain a heat map set of predicted hand bounding boxes;

[0050] The loss function corresponding to the limb capture network to be trained is determined based on the predicted hand bounding box heatmap and the labeled hand bounding box.

[0051] The parameters of the limb capture network to be trained are optimized according to the loss function to obtain the limb capture network.

[0052] Optionally, the predicted hand bounding box heatmap set includes a predicted hand bounding box heatmap, a predicted hand bounding box width and height heatmap, and a predicted hand bounding box offset heatmap.

[0053] The step of determining the loss function of the limb capture network based on the predicted hand bounding box heatmap and the labeled hand bounding boxes includes:

[0054] The first loss function is determined based on the predicted hand bounding box heatmap and the coordinates of the center position of the marked hand bounding box;

[0055] The second loss function is determined based on the predicted hand bounding box width and height heatmap and the fourth width and fourth height values ​​of the hand bounding box.

[0056] The third loss function is determined based on the predicted hand bounding box offset heatmap and the coordinates of the center position of the marked hand bounding box;

[0057] The loss function is constructed based on the first loss function, the second loss function, and the third loss function.

[0058] Optionally, determining the first loss function based on the predicted hand bounding box heatmap and the coordinates of the center position of the labeled hand bounding box includes:

[0059] The supervision function for the predicted hand bounding box heatmap is determined based on the center coordinates of the labeled hand bounding box.

[0060] Obtain the first difference between the predicted hand bounding box heatmap and the supervision function;

[0061] The first loss function is determined based on the first difference.

[0062] Optionally, determining the second loss function based on the predicted hand bounding box width and height heatmap and the labeled width and height of the hand bounding box includes:

[0063] Obtain the fifth width and fifth height values ​​of the predicted hand bounding box heatmap at the center position of the marked hand bounding box;

[0064] Calculate the second difference between the fifth width value and the fourth width value, and calculate the third difference between the fifth height value and the fourth height value;

[0065] The second loss function is determined based on the second difference and the third difference.

[0066] Optionally, determining the third loss function based on the predicted hand bounding box offset heatmap and the coordinates of the center position of the labeled hand bounding box includes:

[0067] The coordinates of the center position of the marked hand bounding box are rounded to obtain the rounded result.

[0068] Obtain the third bias value of the predicted hand bounding box bias heatmap at the rounded result;

[0069] Calculate the fourth difference between the center position coordinates of the marked hand bounding box and the rounded result;

[0070] Calculate the fifth difference between the third bias value and the fourth difference;

[0071] The third loss function is determined based on the fifth difference.

[0072] Optionally, after inputting the human body region image into the limb capture network to obtain the hand bounding box and limb motion, the method further includes:

[0073] Obtain the third confidence level of the left bounding box and the fourth confidence level of the right bounding box respectively;

[0074] If both the third confidence level and the fourth confidence level are greater than or equal to the preset confidence threshold, then the intersection-union ratio of the left-hand bounding box and the right-hand bounding box is obtained;

[0075] If the cross-union ratio is greater than or equal to a preset cross-union ratio threshold, then the two-hand bounding boxes are obtained based on the left-hand bounding box and the right-hand bounding box.

[0076] Optionally, obtaining the hand movement based on the hand bounding box includes:

[0077] The human body region image is cropped using the hand bounding box to obtain a hand region image;

[0078] The hand region image is input into a hand capture network, which extracts features from the hand region image to obtain the hand movement.

[0079] Optionally, obtaining the human body region image from the image to be identified includes:

[0080] The image to be identified is input into a human detection network, and human detection is performed on the image to be identified by the human detection network to obtain a human bounding box;

[0081] The human body bounding box is used to crop the image to be identified to obtain the human body region image.

[0082] According to a second aspect of the present disclosure, a motion capture device is provided, comprising:

[0083] The human body region image acquisition module is configured to acquire human body region images from the image to be identified;

[0084] The hand bounding box acquisition module is configured to input the human body region image into the limb capture network to acquire the hand bounding box and limb motion;

[0085] The hand motion acquisition module is configured to acquire hand motion based on the hand bounding box;

[0086] The human motion acquisition module is configured to fuse the hand movements and the limb movements to obtain human motion.

[0087] Optionally, the limb capture network includes a backbone network, a deconvolution layer connected to the backbone network, a first convolution layer connected to the deconvolution layer, and a hand detection branch network connected to the deconvolution layer.

[0088] The hand bounding box acquisition module is configured as follows:

[0089] The human body region image is input into the backbone network to extract deep features from the human body region image;

[0090] The deep features are input into the deconvolution layer to obtain high-resolution features of the human body region image from the deep features;

[0091] The high-resolution features are input into the first convolutional layer to obtain a two-dimensional key point heatmap of the human body region image;

[0092] The two-dimensional key point heatmap and the high-resolution features are input into the hand detection branch network to obtain the hand bounding box heatmap set of the human body region image;

[0093] The hand bounding frame is determined based on the hand bounding frame heatmap.

[0094] Optionally, the limb capture network further includes a fully connected layer connected to the backbone network;

[0095] The hand bounding box acquisition module is configured to input the deep features into the fully connected layer to obtain the skeletal rotation parameters of the human body region image;

[0096] The limb movement is determined based on the bone rotation parameters and the two-dimensional key point heatmap.

[0097] Optionally, the hand detection branch network includes a second convolutional layer connected to the deconvolutional layer, a splicing unit connected to the second convolutional layer, and a third convolutional layer connected to the splicing unit;

[0098] The hand bounding box acquisition module is configured as follows:

[0099] The high-resolution features are input into the second convolutional layer to obtain the hand bounding box features of the human body region image;

[0100] The hand bounding box feature and the high-resolution feature are input into the stitching unit, and the hand bounding box feature and the high-resolution feature are stitched together to obtain the stitching result;

[0101] The stitching result is input into the third convolutional layer to obtain the hand bounding box heatmap.

[0102] Optionally, the hand bounding box acquisition module is configured as follows:

[0103] The first confidence level of the wrist key points is obtained from the two-dimensional key point heat map;

[0104] The presence of a wrist is determined based on the first confidence level, and the bounding box of the hand is determined based on the determination result.

[0105] Optionally, the hand bounding box heatmap atlas includes a hand bounding box heatmap, a hand bounding box width and height heatmap, and a hand bounding box offset heatmap.

[0106] The hand bounding box acquisition module is configured as follows:

[0107] When it is determined that both the left and right wrists exist or neither exists, obtain the first maximum value of the probability of the hand bounding box existing in the heatmap of the hand bounding box, the first position coordinates corresponding to the first maximum value, and the second confidence level.

[0108] If the second confidence level is greater than or equal to the preset confidence threshold, then based on the first position coordinates, the first width value, the first height value, and the first offset value of the corresponding hand bounding box are obtained from the hand bounding box width and height heat map and the hand bounding box offset heat map, respectively.

[0109] The hand bounding box is determined based on the first width value, the first height value, and the first offset value.

[0110] Optionally, the hand bounding box heatmap atlas includes a hand bounding box heatmap, a hand bounding box width and height heatmap, and a hand bounding box offset heatmap.

[0111] The hand bounding box acquisition module is configured as follows:

[0112] When it is determined that only the left wrist or the right wrist exists, extract the bounding box information of the hand corresponding to the existing wrist;

[0113] Based on the hand bounding box information, a suppression operation is performed on the heatmap of the target hand bounding box corresponding to the non-existent wrist, and the hand bounding box is determined based on the suppression operation result.

[0114] Optionally, the hand bounding box information includes the center position coordinates of the hand bounding box, a second width value, and a second height value;

[0115] The hand bounding box acquisition module is configured as follows:

[0116] Obtain the minimum value between the second height value and the second width value, and determine the target area in the target hand bounding box heat map based on the minimum value and the center position coordinates;

[0117] The probability values ​​corresponding to each location coordinate within the target area are weighted to obtain the suppression operation result;

[0118] Obtain the second maximum value of the probability of a hand bounding box existing in the suppression operation result, the second position coordinates corresponding to the second maximum value, and the fourth confidence level;

[0119] If the fourth confidence level is greater than or equal to the preset confidence threshold, then based on the second position coordinates, the third width value, the third height value, and the second offset value of the corresponding hand bounding box are obtained from the hand bounding box width and height heat map and the hand bounding box offset heat map, respectively.

[0120] The bounding box of the hand corresponding to the non-existent wrist is determined based on the third width value, the third height value, and the second offset value.

[0121] Optionally, the device further includes:

[0122] The limb capture network training module is configured as follows:

[0123] Obtain a training set and train the limb capture network to be trained based on the training set to obtain the limb capture network;

[0124] The training set includes human body region image samples and labeled hand bounding boxes corresponding to the human body region image samples.

[0125] Optionally, the limb capture network training module is configured as follows:

[0126] The human body region image samples are input into the limb capture network to be trained to obtain a heat map set of predicted hand bounding boxes;

[0127] The loss function corresponding to the limb capture network to be trained is determined based on the predicted hand bounding box heatmap and the labeled hand bounding box.

[0128] The parameters of the limb capture network to be trained are optimized according to the loss function to obtain the limb capture network.

[0129] Optionally, the predicted hand bounding box heatmap set includes a predicted hand bounding box heatmap, a predicted hand bounding box width and height heatmap, and a predicted hand bounding box offset heatmap.

[0130] The limb capture network training module is configured as follows:

[0131] The first loss function is determined based on the predicted hand bounding box heatmap and the coordinates of the center position of the marked hand bounding box;

[0132] The second loss function is determined based on the predicted hand bounding box width and height heatmap and the fourth width and fourth height values ​​of the hand bounding box.

[0133] The third loss function is determined based on the predicted hand bounding box offset heatmap and the coordinates of the center position of the marked hand bounding box;

[0134] The loss function is constructed based on the first loss function, the second loss function, and the third loss function.

[0135] Optionally, the limb capture network training module is configured as follows:

[0136] The supervision function for the predicted hand bounding box heatmap is determined based on the center coordinates of the labeled hand bounding box.

[0137] Obtain the first difference between the predicted hand bounding box heatmap and the supervision function;

[0138] The first loss function is determined based on the first difference.

[0139] Optionally, the limb capture network training module is configured as follows:

[0140] Obtain the fifth width and fifth height values ​​of the predicted hand bounding box heatmap at the center position of the marked hand bounding box;

[0141] Calculate the second difference between the fifth width value and the fourth width value, and calculate the third difference between the fifth height value and the fourth height value;

[0142] The second loss function is determined based on the second difference and the third difference.

[0143] Optionally, the limb capture network training module is configured as follows:

[0144] The coordinates of the center position of the marked hand bounding box are rounded to obtain the rounded result.

[0145] Obtain the third bias value of the predicted hand bounding box bias heatmap at the rounded result;

[0146] Calculate the fourth difference between the center position coordinates of the marked hand bounding box and the rounded result;

[0147] Calculate the fifth difference between the third bias value and the fourth difference;

[0148] The third loss function is determined based on the fifth difference.

[0149] Optionally, the device further includes:

[0150] The module for capturing bounding boxes with both hands is configured as follows:

[0151] Obtain the third confidence level of the left bounding box and the fourth confidence level of the right bounding box respectively;

[0152] If both the third confidence level and the fourth confidence level are greater than or equal to the preset confidence threshold, then the intersection-union ratio of the left-hand bounding box and the right-hand bounding box is obtained;

[0153] If the cross-union ratio is greater than or equal to a preset cross-union ratio threshold, then the two-hand bounding boxes are obtained based on the left-hand bounding box and the right-hand bounding box.

[0154] Optionally, the hand motion module is configured as follows:

[0155] The human body region image is cropped using the hand bounding box to obtain a hand region image;

[0156] The hand region image is input into a hand capture network, which extracts features from the hand region image to obtain the hand movement.

[0157] Optionally, the human body region image acquisition module is configured as follows:

[0158] The image to be identified is input into a human detection network, and human detection is performed on the image to be identified by the human detection network to obtain a human bounding box;

[0159] The human body bounding box is used to crop the image to be identified to obtain the human body region image.

[0160] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0161] processor;

[0162] Memory used to store the processor's executable instructions;

[0163] The processor is configured to execute the instructions to implement the motion capture method as described in any one of the first aspects.

[0164] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the motion capture method as described in any one of the first aspects.

[0165] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the motion capture method as described in the first aspect.

[0166] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0167] The motion capture method, apparatus, device, and medium provided in this disclosure involve: acquiring a human body region image from an image to be recognized; inputting the human body region image into a limb capture network to obtain a hand bounding box and limb movements; obtaining hand movements based on the hand bounding box; and fusing the hand movements and limb movements to obtain human limb movements. During motion capture, by inputting the human body region image into the limb capture network, the hand bounding box can be obtained. Since the proportion of the hand region in the human body region image is much higher than its proportion in the entire image to be recognized, the detection rate of the hand bounding box can be greatly improved. Furthermore, only the human body region image needs to be recognized to obtain the hand bounding box, eliminating the need to recognize the entire image to be recognized, thus greatly reducing the computational load. Moreover, only the limb capture network is needed to obtain the hand bounding box and limb movements; no additional hand detection network is required to obtain the hand bounding box, reducing the consumption of computational resources and thereby improving the frame rate of motion capture.

[0168] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0169] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0170] Figure 1 This is a schematic diagram illustrating a motion capture process according to an exemplary embodiment;

[0171] Figure 2 This is a schematic diagram of a limb capture network according to an exemplary embodiment. Figure 1 ;

[0172] Figure 3 This is a schematic architecture diagram of a motion capture system according to an exemplary embodiment;

[0173] Figure 4 This is a flowchart illustrating a motion capture method according to an exemplary embodiment;

[0174] Figure 5 This is a schematic architecture diagram of a motion capture model according to an exemplary embodiment;

[0175] Figure 6 This is a schematic diagram of a limb capture network according to an exemplary embodiment. Figure 2 ;

[0176] Figure 7 This is a flowchart illustrating a method for obtaining a hand bounding box according to an exemplary embodiment;

[0177] Figure 8 This is a flowchart illustrating a method for obtaining a two-handed bounding box according to an exemplary embodiment;

[0178] Figure 9 This is a schematic diagram illustrating a motion capture device according to an exemplary embodiment;

[0179] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0180] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0181] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0182] In related technologies, a separate hand detection network is typically used to obtain the hand bounding box during motion capture. For example... Figure 1 As shown, the images to be recognized are first input into the human detection network and the hand detection network to obtain human bounding boxes and hand bounding boxes, respectively. Then, the human bounding boxes and hand bounding boxes are cropped to obtain human region images and hand region images. Then, the human region images and hand region images are input into the limb capture network and the hand capture network, respectively, to obtain limb movements and hand movements.

[0183] from Figure 1 It can be observed that obtaining the hand bounding box requires recognizing the entire image to be identified, which results in a large computational load. Furthermore, since the hand region occupies a small area in the image, the detection rate is low when using a separate hand detection network to obtain the hand bounding box, and running the hand detection network requires significant computational resources, leading to a decrease in the frame rate of motion capture. Simultaneously, the acquisition of limb movements and hand movements are implemented using different capture networks, resulting in a large data processing volume and low motion capture efficiency in existing motion capture systems.

[0184] Figure 2 A schematic diagram illustrating the structure of a limb capture network in related technologies is shown, such as... Figure 2 As shown, it includes: a backbone network 201, a fully connected layer 202 connected to the backbone network 201, a deconvolutional layer 203 connected to the backbone network 201, and a convolutional layer 204 connected to the deconvolutional layer 203. When acquiring limb movements, the input image to be recognized is processed by the backbone network 201 to extract deep features. These deep features are then regressed through the fully connected layer 202 to obtain skeletal rotation parameters. Simultaneously, the deep features are processed by the deconvolutional layer 203 to obtain high-resolution features, which are then further processed by several convolutional layers 204 for feature extraction, resulting in a two-dimensional human body keypoint heatmap. Finally, by combining the two-dimensional keypoint heatmap and the skeletal rotation parameters, the limb movements can be obtained.

[0185] After capturing hand movements through a hand capture network, the hand movements and limb movements can be stitched together and fused to obtain the capture results of human body movements.

[0186] In view of the above problems, the exemplary embodiments of this disclosure provide a motion capture method, the application scenarios of which include, but are not limited to: live entertainment, human-computer interaction and virtual content creation.

[0187] In one optional implementation, the motion capture function of the terminal device requires support from the motion capture service provider. In order to implement the above-mentioned motion capture method, the exemplary implementation of this disclosure provides a motion capture system. Figure 3 A schematic architecture diagram of the motion capture system is shown. Figure 3 As shown, the motion capture system 300 may include a server 310 and a terminal device 320. The server 310 is a backend server deployed by the motion capture service provider. The terminal device 320 is a terminal device equipped with a hand detection product capable of recognizing hand movements; more specifically, the terminal device may be a smartphone, personal computer, tablet computer, etc. The server 310 and the terminal device 320 can establish a connection via a network to achieve motion capture.

[0188] It should be understood that server 310 can be a single server or a cluster of multiple servers. This disclosure does not limit the specific architecture of server 310.

[0189] The terminal device 320 can acquire an image to be recognized and send it to the server 310. The server 310 can acquire a human body region image from the image to be recognized. The human body region image is input into a limb capture network to obtain a hand bounding box and limb movements. The hand movements are determined based on the hand bounding box, and the hand movements and limb movements are returned to the terminal device 320. The terminal device 320 then fuses the hand movements and limb movements to obtain the human body movements.

[0190] In an optional implementation, the terminal device 320 can also acquire the image to be recognized and send it to the server 310. The server 310 can then acquire a human body region image from the image to be recognized. The human body region image is input into a limb capture network to obtain a hand bounding box and limb movements. The hand movements are determined based on the hand bounding box, and the hand movements and limb movements are fused to obtain a human body movement. The human body movement is then returned to the terminal device 320.

[0191] In another optional implementation, the hand motion recognition function of the terminal device can be implemented independently, thus allowing the terminal device to independently implement the motion capture method provided in this embodiment. Specifically, the terminal device 320 first acquires an image to be recognized and obtains a human body region image from the image to be recognized; then, it inputs the human body region image into a limb capture network to obtain a hand bounding box and limb motion; then, it determines the hand motion based on the hand bounding box; finally, it fuses the hand motion and the limb motion to obtain the human body motion.

[0192] Figure 4 This is a flowchart illustrating a motion capture method according to an exemplary embodiment, such as... Figure 4 As shown, this motion capture method can be applied to servers or terminal devices, such as... Figure 4 As shown, the process includes steps S41 to S44.

[0193] S41. Obtain the human body region image from the image to be identified;

[0194] S42. Input the human body region image into the limb capture network to obtain the hand bounding box and limb movements;

[0195] S43. Obtain hand movements based on the hand bounding box;

[0196] S44. The hand movements and limb movements are fused together to obtain human body movements.

[0197] The method disclosed herein involves: acquiring a human body region image from an image to be recognized; inputting the human body region image into a limb capture network to obtain a hand bounding box and limb motion; obtaining the hand motion based on the hand bounding box; and fusing the hand motion and limb motion to obtain a human limb motion. During motion capture, by inputting the human body region image into the limb capture network, the hand bounding box can be obtained. Since the proportion of the hand region in the human body region image is much higher than its proportion in the entire image to be recognized, the detection rate of the hand bounding box can be greatly improved. Furthermore, only the human body region image needs to be recognized to obtain the hand bounding box, eliminating the need to recognize the entire image to be recognized, thus significantly reducing the computational load. Moreover, only the limb capture network is needed to obtain the hand bounding box and limb motion; no additional hand detection network is required to obtain the hand bounding box, reducing the consumption of computational resources and thereby improving the frame rate of motion capture.

[0198] The following is about Figure 4 Each step in the process will be explained in detail:

[0199] In step S41, a human body region image is obtained from the image to be identified;

[0200] In embodiments of this disclosure, Figure 5 An architecture diagram of a limb capture model is shown, such as... Figure 5 As shown, the limb capture model includes a human detection network 501, a first image cropping unit 502 connected to the human detection network 501, a limb capture network 503 connected to the first image cropping unit 502, a second image cropping unit 504 connected to the limb capture network 503, a hand capture network 505 connected to the second image cropping unit 504, and a stitching and fusion unit 506 connected to the limb capture network 503 and the hand capture network 505. When executing the motion capture method in this embodiment, firstly... Figure 5 The human detection network 501 and the first image cropping unit 502 process the image to be identified to obtain a human region image. Specifically, the image to be identified is first input into the human detection network 501 to obtain a human bounding box; then, the human bounding box is input into the first image cropping unit 502 so that the first image cropping unit 502 uses the human bounding box to crop the image to be identified to obtain a human region image located within the human bounding box. The human detection network 503 is a neural network model pre-trained using image samples labeled with human bounding boxes.

[0201] In step S42, the human body region image is input into the limb capture network to obtain the hand bounding box and limb motion.

[0202] In embodiments of this disclosure, Figure 6 The schematic diagram illustrates the structure of a limb capture network, such as Figure 6 As shown, the limb capture network includes a backbone network 601, a deconvolution layer 602 connected to the backbone network 601, a first convolutional layer 603 connected to the deconvolution layer 602, and a hand detection branch 604 connected to the first convolutional layer 603. Relative to... Figure 2 The limb capture network structure shown in the related technologies is different from that in this application. The limb capture network adds a hand detection branch. While acquiring limb movements, it can simultaneously obtain a hand bounding box heatmap based on the hand detection branch, and then obtain the hand bounding box from the heatmap. Therefore, the limb capture network of this application can simultaneously acquire the hand bounding box and limb movements, eliminating the need for an additional hand detection network to acquire the hand bounding box. This reduces computational resource consumption and improves the frame rate of motion capture.

[0203] Next, based on Figure 6 The schematic diagram of the limb capture network shown illustrates in detail how the hand bounding box and limb movements are obtained in the embodiments of this application.

[0204] Figure 7 This schematically illustrates a process for obtaining a hand bounding box, such as... Figure 7 As shown, the process includes at least:

[0205] S421. Input the human body region image into the backbone network to extract the deep features of the human body region image;

[0206] S422. Input the deep features into the deconvolution layer to obtain high-resolution features of the human body region image from the deep features;

[0207] S423. Input the high-resolution features into the first convolutional layer to obtain a two-dimensional key point heatmap of the human body region image;

[0208] S424. Input the two-dimensional key point heatmap and the high-resolution features into the hand detection branch network to obtain the hand bounding box heatmap set of the human body region image;

[0209] S425. Determine the hand bounding frame based on the hand bounding frame heat map.

[0210] Specifically, the human body region image is input into the backbone network 601, which extracts deep features from the human body region image. These deep features are then passed through a deconvolution layer 602 to obtain high-resolution features. These high-resolution features are then processed through several first convolutional layers 603 for feature extraction, resulting in a two-dimensional keypoint heatmap. Simultaneously, the high-resolution features and the two-dimensional keypoint heatmap are input into the hand detection branch 604 to obtain a hand bounding box heatmap set of the human body region image. The hand bounding box is then determined based on the hand bounding box heatmap set. The backbone network can be a convolutional neural network, or other neural networks built upon convolutional neural networks for extracting image features; this embodiment does not impose specific limitations.

[0211] In embodiments of this disclosure, such as Figure 6 As shown, the hand detection branch network 604 includes a second convolutional layer 6041 connected to the deconvolutional layer 603, a splicing unit 6042 connected to the second convolutional layer 6041, and a third convolutional layer 6043 connected to the splicing unit 6042.

[0212] After inputting the two-dimensional keypoint heatmap and the high-resolution feature into the hand detection branch network, the hand bounding box heatmap set in the human body region image can be obtained according to the following procedure, which includes at least the following:

[0213] S4241. Input the high-resolution features into the second convolutional layer to obtain the hand bounding box features of the human body region image;

[0214] S4242. Input the hand bounding box feature and the two-dimensional key point heat map into the stitching unit, and perform a stitching operation on the hand bounding box feature and the two-dimensional key point heat map to obtain the stitching result;

[0215] S4243. Input the stitching result into the third convolutional layer to obtain the hand bounding box heatmap.

[0216] For example, high-resolution features are input into the second convolutional layer 6041, where feature extraction is performed to obtain the hand bounding box features. The hand bounding box features and a 2D keypoint heatmap are then input into the stitching unit 6042, where they are stitched together to obtain the stitched result. Finally, the stitched result is passed through the third convolutional layer 6043 for feature extraction to obtain the hand bounding box heatmap.

[0217] In embodiments of this disclosure, the size of the input image to be recognized can be... After feature extraction through the human detection network 501, the first image cropping unit 502, the backbone network 601, and the deconvolution layer 602, a value of [size missing] can be obtained. High-resolution features are then extracted using the first convolutional layer 603, yielding a feature size of [missing value]. A two-dimensional keypoint heatmap is generated, and high-resolution features are extracted through the second convolutional layer 6041 in the hand detection branch network 604, resulting in a size of [missing information]. The hand bounding box feature is then processed by the stitching unit 6042 to obtain a size of [size missing]. The features are then processed by a third convolutional layer (6043) to extract features from the concatenated features, thus obtaining... of .in, ,and Number of samples selected for one training session of the limb capture network N represents the number of channels in the heatmap of the final output hand bounding box heatmap set. N is an even number, meaning that half of the channels correspond to the deep channel features of the left hand, and the other half correspond to the deep channel features of the right hand. For example... If it can be 10, then the heatmap in the hand bounding box heatmap set includes 5 channels of deep features for each of the left and right hands. , These are the width and height of the image to be recognized, respectively.

[0218] In embodiments of this disclosure, such as Figure 6 As shown, the limb capture network also includes a fully connected layer 605 connected to the backbone network 601. By inputting the deep features output by the backbone network into the fully connected layer, limb movements can be captured. Specifically:

[0219] S426. Input the deep features into the fully connected layer to obtain the skeletal rotation parameters of the human body region image;

[0220] S427. Determine the limb movement based on the bone rotation parameters and the two-dimensional key point heat map.

[0221] The limb capture network disclosed herein extracts features from a high-resolution feature pass through several convolutional layers in the hand detection branch, and then concatenates the feature extraction results with a two-dimensional keypoint heatmap to predict the hand bounding box heatmaps for both the left and right hands. Furthermore, since the two-dimensional keypoint heatmap contains the positional information of the left and right wrists, it provides a strong prior for predicting the hand bounding box, thus improving the detection accuracy. Simultaneously, by reusing deconvolutional layers when acquiring the hand bounding box and the two-dimensional keypoint heatmap, the computational cost of the limb capture network can be significantly reduced.

[0222] In embodiments of this disclosure, determining the hand bounding box based on the hand bounding box heatmap includes:

[0223] S4251. Obtain the first confidence level of the wrist key points from the two-dimensional key point heat map;

[0224] S4252. Determine whether the wrist exists based on the first confidence level, and determine the bounding box of the hand based on the determination result.

[0225] For example, a two-dimensional keypoint heatmap includes the coordinates and confidence levels of two-dimensional keypoints on the human body. Wrist keypoints can be determined from the two-dimensional keypoint heatmap, and a first confidence level for the wrist keypoints can be obtained. If the first preset confidence threshold is... When the first confidence level is less than When the first confidence level is greater than or equal to the specified level, it is determined that the wrist does not exist; when the first confidence level is greater than or equal to the specified level, it is determined that the wrist does not exist. At that time, it was determined that the wrist was present.

[0226] In embodiments of this disclosure, the hand bounding box heatmap atlas includes a hand bounding box heatmap, a hand bounding box width and height heatmap, and a hand bounding box offset heatmap; determining the hand bounding box based on the judgment result includes:

[0227] When it is determined that both the left and right wrists exist or neither exists, the first maximum value of the probability of the hand bounding box existing in the hand bounding box heatmap, the first position coordinates corresponding to the first maximum value, and the second confidence level are obtained; if the second confidence level is greater than or equal to a preset confidence threshold, the first width value, the first height value, and the first offset value of the corresponding hand bounding box are obtained from the hand bounding box width and height heatmap and the hand bounding box offset heatmap based on the first position coordinates; the hand bounding box is determined according to the first width value, the first height value, and the first offset value.

[0228] Specifically, if it is determined that both the left and right wrists exist or neither exists, the heatmap of the hand bounding box is first obtained. The first maximum value in and the first maximum value Position coordinates ( c, c) If the first maximum value is less than the second preset confidence threshold (For example, 0.5), then from the heatmap of the hand's bounding box width and height. Get from The third width value at the location wc and the third height value h and the third width value wc and the third height value h Then, from the hand-bound frame offset heatmap Get from Position coordinates The second bias value at the location and Furthermore, based on the third width value wc、 Third height value h and the second bias value and The coordinates of the center position of the hand bounding box are: The width and height of the hand's bounding box are respectively wc and h Confidence level is .

[0229] When it is determined that only the left wrist or the right wrist exists, the bounding box information of the hand corresponding to the existing wrist is extracted; based on the bounding box information, the heatmap of the bounding box of the target hand corresponding to the non-existent wrist is suppressed, and the bounding box of the hand is determined based on the suppression operation result.

[0230] For example, if the confidence level of the wrist key points determines that the left wrist exists and the right wrist does not, then the left-hand bounding box information is first obtained. Then, the heatmap of the right-hand bounding box is suppressed based on the left-hand bounding box information, and the right-hand bounding box is determined based on the suppression result. The method for obtaining the left-hand bounding box information can be referred to in the above embodiment, and will not be repeated here.

[0231] In embodiments of this disclosure, the hand bounding box information includes the center coordinates of the hand bounding box, a second width value, and a second height value; the step of performing a suppression operation on the heatmap of the target hand bounding box corresponding to a non-existent wrist based on the hand bounding box information, and determining the hand bounding box based on the suppression operation result, includes:

[0232] The minimum value between the second height value and the second width value is obtained, and the target region is determined in the target hand bounding box heatmap based on the minimum value and the center position coordinates. The probability values ​​corresponding to each position coordinate in the target region are weighted to obtain the suppression operation result. The second maximum value of the probability of the existence of the hand bounding box in the suppression operation result, the second position coordinates corresponding to the second maximum value, and the fourth confidence level are obtained. If the fourth confidence level is greater than or equal to the preset confidence threshold, the third width value, the third height value, and the second offset value of the corresponding hand bounding box are obtained from the hand bounding box width and height heatmap and the hand bounding box offset heatmap based on the second position coordinates. The hand bounding box corresponding to the non-existent wrist is determined based on the third width value, the third height value, and the second offset value.

[0233] Specifically, the left hand bounding box information includes the coordinates of the center position of the left hand bounding box. Second width value and the second height value First, we can start with the second width value. Second height value min In the right-hand bounding box heatmap and Next, the probability values ​​corresponding to the coordinates of each location within the target area of ​​the right-hand bounding box heatmap are weighted to obtain the suppression operation result. Specifically, the probability values ​​corresponding to the coordinates of each location within the target area of ​​the right-hand bounding box heatmap are multiplied by a coefficient. .in Let d be the variance of the Gaussian distribution corresponding to the right-hand bounding box heatmap, and d be the coordinates of each position in the right-hand bounding box heatmap. The distance. Further, after obtaining the suppression operation result, the second maximum value of the probability of the hand bounding box existing in the suppression operation result, the second position coordinates corresponding to the second maximum value, and the fourth confidence level are obtained; if the fourth confidence level is greater than or equal to the preset confidence threshold, the third width value, the third height value, and the second offset value of the right hand bounding box are obtained from the right hand bounding box width and height heatmap and the right hand bounding box offset heatmap respectively based on the second position coordinates; the right hand bounding box is determined according to the third width value, the third height value, and the second offset value.

[0234] Because the image features of the left and right hands are quite similar, for example, when the left hand is in the frame but the right hand is not, the maximum value of the heatmap of the right hand bounding box is... If the value is greater than or equal to the second preset confidence threshold and its position is near the left hand, the right-hand bounding box will be misdetected as the left-hand bounding box, resulting in a low accuracy rate for detecting the right-hand bounding box. This leads to the problem of misdetecting both left-hand and right-hand bounding boxes. However, by performing a suppression operation on the heatmap of the target hand bounding box corresponding to the non-existent wrist based on the hand bounding box information, and then determining the hand bounding box corresponding to the non-existent wrist based on the suppression operation result, the problem of low accuracy for hand bounding boxes corresponding to non-existent wrists can be effectively avoided, thus improving the accuracy of hand bounding boxes corresponding to non-existent wrists.

[0235] In embodiments of this disclosure, before inputting the human body region image into the limb capture network, the limb capture network to be trained needs to be trained to obtain a stable limb capture network. Specifically, a training set can be obtained, and the limb capture network to be trained can be trained based on the training set to obtain the limb capture network; wherein, the training set includes human body region image samples and labeled hand bounding boxes corresponding to the human body region image samples.

[0236] The annotation of the hand bounding box includes annotated left-hand bounding box and annotated right-hand bounding box. The limb capture network to be trained is trained using a training set that includes human body region image samples and annotated left-hand bounding boxes and annotated right-hand bounding boxes corresponding to the human body region image samples. This can improve the accuracy of the limb capture network in obtaining left-hand bounding boxes and right-hand bounding boxes from human body region images.

[0237] In embodiments of this disclosure, training the limb capture network to be trained based on the training set to obtain the limb capture network includes:

[0238] S451. Input the human body region image samples into the limb capture network to be trained to obtain a heat map set of predicted hand bounding boxes;

[0239] In embodiments of this disclosure, the predicted hand bounding box heatmap set includes a predicted hand bounding box heatmap, a predicted hand bounding box width and height heatmap, and a predicted hand bounding box offset heatmap.

[0240] S452. Determine the loss function corresponding to the limb capture network to be trained based on the predicted hand bounding box heatmap and the labeled hand bounding box;

[0241] In the embodiments of this disclosure, a first loss function can be determined based on the predicted hand bounding box heatmap and the center coordinates of the labeled hand bounding box; a second loss function can be determined based on the predicted hand bounding box width and height heatmap and the fourth width and fourth height values ​​of the labeled hand bounding box; a third loss function can be determined based on the predicted hand bounding box offset heatmap and the center coordinates of the labeled hand bounding box; finally, a loss function corresponding to the limb capture network to be trained can be constructed based on the first loss function, the second loss function, and the third loss function.

[0242] The following describes, with reference to specific embodiments, how to determine the first loss function based on the predicted hand bounding box heatmap and the coordinates of the center position of the labeled hand bounding box.

[0243] In embodiments of this disclosure, the terminal device can determine the supervision function of the predicted hand bounding box heatmap based on the center position coordinates of the labeled hand bounding box; obtain a first difference between the predicted hand bounding box heatmap and the supervision function; and determine the first loss function based on the first difference.

[0244] For example, the coordinates of the center position of the bounding box of the hand are: Predicting the hand bounding box heatmap For single-channel prediction, predict the hand bounding box heatmap. each point The value at that point represents the probability that a hand bounding box exists at that point. The supervision function for determining the predicted hand bounding box heatmap is determined based on the coordinates of the center position of the labeled hand bounding box. Therefore Centered on, variance is The Gaussian distribution is used. Further, the predicted hand bounding box heatmap is calculated. Supervision function First difference The first loss function is shown in formula (1):

[0245] S 1= (1)

[0246] Wherein, S1 is the first loss function.

[0247] The first loss function is determined based on the predicted hand bounding box heatmap and the coordinates of the center position of the labeled hand bounding box. Then, the limb capture network is trained based on the loss function of the limb network constructed based on the first loss function. This enables the limb capture network to accurately output the hand bounding box heatmap from the human body region image, and thus accurately determine the hand bounding box based on the hand bounding box heatmap.

[0248] The following describes, with reference to specific embodiments, how to determine the second loss function based on the predicted hand bounding box width and height heatmap and the labeled width and height of the hand bounding box.

[0249] In embodiments of this disclosure, the terminal device can obtain the fifth width value and the fifth height value of the predicted hand bounding box heatmap at the center position of the marked hand bounding box; calculate a second difference between the fifth width value and the fourth width value, and calculate a third difference between the fifth height value and the fourth height value; and determine the second loss function based on the second difference and the third difference.

[0250] For example, the fourth width value and the fourth height value of the bounding box of the hand are respectively and Predicting the width and height of the hand bounding box using a heatmap. Two channels are used to predict the width and height of the hand bounding box in a heatmap. each point The value at that point represents the width and height of the hand bounding box at that point. When obtaining the second loss function, first mark the center position of the hand bounding box. Perform a rounding operation to obtain the center position after rounding. The fifth width value at the location w and the fifth height value h Further, calculate the fifth width value. w and the fourth width value The second difference Fifth height value h and the fourth height value The third difference Then based on the second difference The calculated second loss function is shown in Equation (2):

[0251] S 2= (2)

[0252] in, S 2 represents the second loss function.

[0253] The second loss function is determined by the predicted width and height heatmap of the hand bounding box and the labeled width and height of the hand bounding box. Then, the limb capture network is trained based on the loss function of the limb network constructed by the second loss function. This enables the limb capture network to accurately output the width and height heatmap of the hand bounding box from the human body region image, and thus accurately determine the hand bounding box based on the width and height heatmap of the hand bounding box.

[0254] The following describes, with reference to specific embodiments, how to determine the third loss function based on the predicted hand bounding box offset heatmap and the coordinates of the center position of the labeled hand bounding box.

[0255] In embodiments of this disclosure, the terminal device can perform a rounding operation on the center position coordinates of the labeled hand bounding box to obtain a rounding result; obtain a third bias value of the predicted hand bounding box bias heatmap at the rounding result; calculate a fourth difference between the center position coordinates of the labeled hand bounding box and the rounding result; calculate a fifth difference between the third bias value and the fourth difference; and determine the third loss function based on the fifth difference.

[0256] For example, the predicted hand bounding box bias heatmap The predicted hand bounding box offset heatmap is a two-channel image. each point The value at that point represents the bias value of the hand bounding box at that point. When determining the third loss function, the following is obtained: exist The third bias value at the location and Recalculate , The loss function is shown in equation (3):

[0257] S 3=

[0258] Wherein, S3 is the third loss function.

[0259] A third loss function is determined based on the predicted hand bounding box offset heatmap and the center coordinates of the labeled hand bounding box. Then, the limb capture network is trained based on the loss function of the limb network constructed based on the third loss function. This enables the limb capture network to accurately output the hand bounding box offset heatmap from the human body region image, and thus accurately determine the hand bounding box based on the hand bounding box offset heatmap.

[0260] Furthermore, after determining the first loss function, the second loss function, and the third loss function, the loss function of the limb capture network is constructed based on these three loss functions. For example, the average value of the first, second, and third loss functions, or the weighted sum of these three loss functions, can be used as the loss function of the limb capture network. Of course, the loss function of the limb capture network can also be other functions constructed from the first, second, and third loss functions; this embodiment does not impose specific limitations here.

[0261] S453. Optimize the parameters of the limb capture network to be trained according to the loss function to obtain the limb capture network.

[0262] For example, during the parameter optimization process of the limb capture network to be trained, the value of the loss function is obtained. When the value of the loss function reaches its minimum, the parameter optimization of the limb capture network to be trained is completed, and the obtained limb capture network can accurately obtain the hand bounding box from the human body region image.

[0263] In the embodiments of this disclosure, the left and right hands in the image to be identified may have interactive actions, so it is impossible to identify the left hand bounding box and the right hand bounding box separately. Motion capture can only be achieved by identifying the bounding boxes of both hands. Figure 8 This schematically illustrates the process of obtaining the bounding box of both hands, as shown below. Figure 8 As shown, the process includes at least:

[0264] S461. Obtain the third confidence level of the left bounding box and the fourth confidence level of the right bounding box respectively;

[0265] S462. If both the third confidence level and the fourth confidence level are greater than or equal to a preset confidence threshold, then obtain the intersection-union ratio of the left-hand bounding box and the right-hand bounding box.

[0266] S463. If the cross-union ratio is greater than or equal to a preset cross-union ratio threshold, then obtain the two-hand bounding box based on the left-hand bounding box and the right-hand bounding box.

[0267] Specifically, when the confidence scores of the obtained left-hand bounding box and right-hand bounding box are respectively and At that time, if and Then calculate the Intersection over Union (IOU) of the bounding boxes of the left and right hands. If the IOU is greater than a preset IOU threshold... When the overlap between the left and right bounding boxes is high, it indicates that this is a two-handed interactive action. Furthermore, the bounding boxes for both hands can be calculated using the following formula:

[0268]

[0269]

[0270]

[0271] (4)

[0272] in, The center of the left-hand bounding box, The central location, The width of the left-hand bounding box. The height of the left-hand bounding box, The width of the hand-shaped frame. The height of the hand-shaped frame.

[0273] By determining the bounding boxes of both hands based on the left-hand bounding box and the bounding box of the right hand when the intersection-union ratio of the left-hand bounding box and the right-hand bounding box is greater than or equal to a preset intersection-union ratio threshold, the bounding boxes of both hands can be output during the interaction of the hands. The bounding boxes of both hands can be cropped to obtain the hand region image, and the interaction of the hands can be accurately identified from the hand region image.

[0274] In step S43, the hand movement is obtained based on the hand bounding box.

[0275] In embodiments of this disclosure, such as Figure 5 As shown, a hand bounding box can be input into the second image cropping unit 504, so that the second image cropping unit 504 uses the hand bounding box to crop the human body region image to obtain a hand region image located within the hand bounding box. Then, this hand region image is input into the hand capture network 505, so that the hand capture network 505 performs feature extraction on the hand region image to obtain the hand movement. The hand capture network is a neural network model pre-trained using image samples labeled with hand bounding boxes and corresponding hand movements.

[0276] In step S44, the hand movements and the limb movements are merged to obtain human body movements.

[0277] In embodiments of this disclosure, such as Figure 5 As shown, hand and limb movements can be input into the stitching and fusion unit 506, which then stitches and fuses the hand and limb movements to obtain human body movements. The purpose of stitching and fusion is to combine the features extracted from the image into a feature that is more discriminative than the input features. Stitching and fusion can typically be performed in two ways: one is to directly stitch the two features together. For example, if the dimensions of the two input features x and y are p and q, then the dimension of the output feature z after feature fusion is p+q; the other is to combine the two feature vectors into a complex vector, where for input features x and y, the fused feature z = x + iy, where i is the imaginary unit.

[0278] In summary, the motion capture method provided in this disclosure not only improves the detection rate of hand bounding boxes, reduces computational load, and increases the motion capture frame rate, but also effectively avoids the problem of low accuracy of hand bounding boxes corresponding to non-existent wrists by suppressing the heatmap of the target hand bounding box corresponding to the non-existent wrist based on the hand bounding box information, and then determining the hand bounding box corresponding to the non-existent wrist based on the suppression operation result. Furthermore, it can output the hand bounding boxes during hand interaction and crop the hand bounding boxes to obtain a hand region image, thereby accurately identifying the hand interaction from the hand region image.

[0279] Figure 9 This is a schematic diagram of a limb capture device according to an exemplary embodiment, such as... Figure 9 As shown, the limb capture device 900 includes:

[0280] The human body region image acquisition module 901 is configured to acquire human body region images from the image to be identified;

[0281] The hand bounding box acquisition module 902 is configured to input the human body region image into the limb capture network to acquire the hand bounding box and limb motion.

[0282] The hand motion acquisition module 903 is configured to acquire hand motion based on the hand bounding box;

[0283] The human motion acquisition module 904 is configured to fuse the hand motion and the limb motion to obtain human motion.

[0284] Optionally, the limb capture network includes a backbone network, a deconvolution layer connected to the backbone network, a first convolution layer connected to the deconvolution layer, and a hand detection branch network connected to the deconvolution layer.

[0285] The hand bounding box acquisition module 902 is configured as follows:

[0286] The human body region image is input into the backbone network to extract deep features from the human body region image;

[0287] The deep features are input into the deconvolution layer to obtain high-resolution features of the human body region image from the deep features;

[0288] The high-resolution features are input into the first convolutional layer to obtain a two-dimensional key point heatmap of the human body region image;

[0289] The two-dimensional key point heatmap and the high-resolution features are input into the hand detection branch network to obtain the hand bounding box heatmap set of the human body region image;

[0290] The hand bounding frame is determined based on the hand bounding frame heatmap.

[0291] Optionally, the limb capture network further includes a fully connected layer connected to the backbone network;

[0292] The hand bounding box acquisition module 902 is configured as follows:

[0293] The deep features are input into the fully connected layer to obtain the skeletal rotation parameters of the human body region image;

[0294] The limb movement is determined based on the bone rotation parameters and the two-dimensional key point heatmap.

[0295] Optionally, the hand detection branch network includes a second convolutional layer connected to the deconvolutional layer, a splicing unit connected to the second convolutional layer, and a third convolutional layer connected to the splicing unit;

[0296] The hand bounding box acquisition module 902 is configured as follows:

[0297] The high-resolution features are input into the second convolutional layer to obtain the hand bounding box features of the human body region image;

[0298] The hand bounding box feature and the high-resolution feature are input into the stitching unit, and the hand bounding box feature and the high-resolution feature are stitched together to obtain the stitching result;

[0299] The stitching result is input into the third convolutional layer to obtain the hand bounding box heatmap.

[0300] Optionally, the hand bounding box acquisition module 902 is configured to:

[0301] The first confidence level of the wrist key points is obtained from the two-dimensional key point heat map;

[0302] The presence of a wrist is determined based on the first confidence level, and the bounding box of the hand is determined based on the determination result.

[0303] Optionally, the hand bounding box heatmap atlas includes a hand bounding box heatmap, a hand bounding box width and height heatmap, and a hand bounding box offset heatmap.

[0304] The hand bounding box acquisition module 902 is configured as follows:

[0305] When it is determined that both the left and right wrists exist or neither exists, obtain the first maximum value of the probability of the hand bounding box existing in the heatmap of the hand bounding box, the first position coordinates corresponding to the first maximum value, and the second confidence level.

[0306] If the second confidence level is greater than or equal to the preset confidence threshold, then based on the first position coordinates, the first width value, the first height value, and the first offset value of the corresponding hand bounding box are obtained from the hand bounding box width and height heat map and the hand bounding box offset heat map, respectively.

[0307] The hand bounding box is determined based on the first width value, the first height value, and the first offset value.

[0308] Optionally, the hand bounding box heatmap atlas includes a hand bounding box heatmap, a hand bounding box width and height heatmap, and a hand bounding box offset heatmap.

[0309] The hand bounding box acquisition module 902 is configured as follows:

[0310] When it is determined that only the left wrist or the right wrist exists, extract the bounding box information of the hand corresponding to the existing wrist;

[0311] Based on the hand bounding box information, a suppression operation is performed on the heatmap of the target hand bounding box corresponding to the non-existent wrist, and the hand bounding box is determined based on the suppression operation result.

[0312] Optionally, the hand bounding box information includes the third confidence level of the hand bounding box, the center position coordinates, the second width value, and the second height value;

[0313] The hand bounding box acquisition module 902 is configured as follows:

[0314] Obtain the minimum value between the second height value and the second width value, and determine the target area in the target hand bounding box heat map based on the minimum value and the center position coordinates;

[0315] The probability values ​​corresponding to each location coordinate within the target area are weighted to obtain the suppression operation result;

[0316] Obtain the second maximum value of the probability of a hand bounding box existing in the suppression operation result, the second position coordinates corresponding to the second maximum value, and the fourth confidence level;

[0317] If the fourth confidence level is greater than or equal to the preset confidence threshold, then based on the second position coordinates, the third width value, the third height value, and the second offset value of the corresponding hand bounding box are obtained from the hand bounding box width and height heat map and the hand bounding box offset heat map, respectively.

[0318] The bounding box of the hand corresponding to the non-existent wrist is determined based on the third width value, the third height value, and the second offset value.

[0319] Optionally, the device further includes:

[0320] The limb capture network training module 905 is configured as follows:

[0321] Obtain a training set and train the limb capture network to be trained based on the training set to obtain the limb capture network;

[0322] The training set includes human body region image samples and labeled hand bounding boxes corresponding to the human body region image samples.

[0323] Optionally, the limb capture network training module 905 is configured as follows:

[0324] The human body region image samples are input into the limb capture network to be trained to obtain a heat map set of predicted hand bounding boxes;

[0325] The loss function corresponding to the limb capture network to be trained is determined based on the predicted hand bounding box heatmap and the labeled hand bounding box.

[0326] The parameters of the limb capture network to be trained are optimized according to the loss function to obtain the limb capture network.

[0327] Optionally, the predicted hand bounding box heatmap set includes a predicted hand bounding box heatmap, a predicted hand bounding box width and height heatmap, and a predicted hand bounding box offset heatmap.

[0328] The limb capture network training module 905 is configured as follows:

[0329] The first loss function is determined based on the predicted hand bounding box heatmap and the coordinates of the center position of the marked hand bounding box;

[0330] The second loss function is determined based on the predicted hand bounding box width and height heatmap and the fourth width and fourth height values ​​of the hand bounding box.

[0331] The third loss function is determined based on the predicted hand bounding box offset heatmap and the coordinates of the center position of the marked hand bounding box;

[0332] The loss function is constructed based on the first loss function, the second loss function, and the third loss function.

[0333] Optionally, the limb capture network training module 905 is configured as follows:

[0334] The supervision function for the predicted hand bounding box heatmap is determined based on the center coordinates of the labeled hand bounding box.

[0335] Obtain the first difference between the predicted hand bounding box heatmap and the supervision function;

[0336] The first loss function is determined based on the first difference.

[0337] Optionally, the limb capture network training module 905 is configured as follows:

[0338] Obtain the fifth width and fifth height values ​​of the predicted hand bounding box heatmap at the center position of the marked hand bounding box;

[0339] Calculate the second difference between the fifth width value and the fourth width value, and calculate the third difference between the fifth height value and the fourth height value;

[0340] The second loss function is determined based on the second difference and the third difference.

[0341] Optionally, the limb capture network training module 905 is configured as follows:

[0342] The coordinates of the center position of the marked hand bounding box are rounded to obtain the rounded result.

[0343] Obtain the third bias value of the predicted hand bounding box bias heatmap at the rounded result;

[0344] Calculate the fourth difference between the center position coordinates of the marked hand bounding box and the rounded result;

[0345] Calculate the fifth difference between the third bias value and the fourth difference;

[0346] The third loss function is determined based on the fifth difference.

[0347] Optionally, the device further includes:

[0348] The double-bound box acquisition module 906 is configured as follows:

[0349] Obtain the third confidence level of the left bounding box and the fourth confidence level of the right bounding box respectively;

[0350] If both the third confidence level and the fourth confidence level are greater than or equal to the preset confidence threshold, then the intersection-union ratio of the left-hand bounding box and the right-hand bounding box is obtained;

[0351] If the cross-union ratio is greater than or equal to a preset cross-union ratio threshold, then the two-hand bounding boxes are obtained based on the left-hand bounding box and the right-hand bounding box.

[0352] Optionally, the hand motion module 903 is configured as follows:

[0353] The human body region image is cropped using the hand bounding box to obtain a hand region image;

[0354] The hand region image is input into a hand capture network, which extracts features from the hand region image to obtain the hand movement.

[0355] Optionally, the human body region image acquisition module 901 is configured as follows:

[0356] The image to be identified is input into a human detection network, and human detection is performed on the image to be identified by the human detection network to obtain a human bounding box;

[0357] The human body bounding box is used to crop the image to be identified to obtain the human body region image.

[0358] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0359] Exemplary embodiments of this disclosure also provide an electronic device, which may be a terminal device. References below... Figure 10 This electronic device will be described. It should be understood that... Figure 10 The electronic device 1000 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0360] like Figure 10 As shown, the electronic device 1000 is presented in the form of a general-purpose computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processing unit 1010, at least one storage unit 1020, and a bus 1030 connecting different system components (including storage unit 1020 and processing unit 1010).

[0361] The storage unit stores program code, which can be executed by the processing unit 1010 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 1010 can perform, as follows: Figure 4 The methods and steps shown are as follows.

[0362] Storage unit 1020 may include volatile storage units, such as random access memory (RAM) 1021 and / or cache memory 1022, and may further include read-only memory (ROM) 1023.

[0363] Storage unit 1020 may also include a program / utility 1024 having a set (at least one) program module 1025, such program module 1025 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0364] Bus 1030 may include a data bus, an address bus, and a control bus.

[0365] Electronic device 1000 can also communicate with one or more external devices 1100 (e.g., keyboards, pointing devices, Bluetooth devices, etc.) via input / output (I / O) interface 1040. Electronic device 1000 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 1050. As shown, network adapter 1050 communicates with other modules of electronic device 1000 via bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0366] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0367] Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be embodied in entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.” Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0368] In addition, this disclosure also provides a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by the processor of an electronic device, enables the electronic device to perform the motion capture method provided in the above embodiments.

[0369] In addition, this disclosure also provides a computer program product, including computer instructions, which, when executed on an electronic device, cause the electronic device to perform the motion capture method provided in the above embodiments.

[0370] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0371] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A motion capture method, characterized in that, include: Obtain human body region images from the image to be identified; The human body region image is input into the limb capture network to obtain the hand bounding box and limb movement. The hand bounding box is determined based on the hand bounding box heat map set, which includes the hand bounding box heat map, the hand bounding box width and height heat map, and the hand bounding box offset heat map. The hand movement is obtained based on the hand bounding box; The hand movements and limb movements are combined to obtain human body movements; The hand-enclosing frame is obtained through the following steps: A two-dimensional key point heatmap of the human body region image is obtained, and the first confidence level of the wrist key point is obtained from the two-dimensional key point heatmap. The presence of a wrist is determined based on the first confidence level, and the bounding box of the hand is determined based on the determination result; Determining the hand bounding box based on the judgment result includes: When it is determined that only the left wrist or the right wrist exists, extract the bounding box information of the hand corresponding to the existing wrist; Based on the hand bounding box information, a suppression operation is performed on the heatmap of the target hand bounding box corresponding to the non-existent wrist, and the hand bounding box is determined based on the suppression operation result.

2. The method according to claim 1, characterized in that, The limb capture network includes a backbone network, a deconvolution layer connected to the backbone network, a first convolution layer connected to the deconvolution layer, and a hand detection branch network connected to the deconvolution layer. The step of inputting the human body region image into the limb capture network to obtain the hand bounding box includes: The human body region image is input into the backbone network to extract deep features from the human body region image; The deep features are input into the deconvolution layer to obtain high-resolution features of the human body region image from the deep features; The high-resolution features are input into the first convolutional layer to obtain a two-dimensional key point heatmap of the human body region image; The two-dimensional key point heatmap and the high-resolution features are input into the hand detection branch network to obtain the hand bounding box heatmap set of the human body region image; The hand bounding frame is determined based on the hand bounding frame heatmap.

3. The method according to claim 2, characterized in that, The limb capture network also includes a fully connected layer connected to the backbone network; The step of inputting the human body region image into the limb capture network to obtain limb movements includes: The deep features are input into the fully connected layer to obtain the skeletal rotation parameters of the human body region image; The limb movement is determined based on the bone rotation parameters and the two-dimensional key point heatmap.

4. The method according to claim 2, characterized in that, The hand detection branch network includes a second convolutional layer connected to the deconvolutional layer, a splicing unit connected to the second convolutional layer, and a third convolutional layer connected to the splicing unit; The step of inputting the two-dimensional keypoint heatmap and the high-resolution features into the hand detection branch network to obtain a heatmap set of hand bounding boxes in the human body region image includes: The high-resolution features are input into the second convolutional layer to obtain the hand bounding box features of the human body region image; The hand bounding box feature and the two-dimensional key point heatmap are input into the stitching unit, and the hand bounding box feature and the two-dimensional key point heatmap are stitched together to obtain the stitching result. The stitching result is input into the third convolutional layer to obtain the hand bounding box heatmap.

5. The method according to claim 1, characterized in that, The hand bounding box heatmap atlas includes a hand bounding box heatmap, a hand bounding box width and height heatmap, and a hand bounding box offset heatmap; Determining the hand bounding box based on the judgment result includes: When it is determined that both the left and right wrists exist or neither exists, obtain the first maximum value of the probability of the hand bounding box existing in the heatmap of the hand bounding box, the first position coordinates corresponding to the first maximum value, and the second confidence level. If the second confidence level is greater than or equal to the preset confidence threshold, then based on the first position coordinates, the first width value, the first height value, and the first offset value of the corresponding hand bounding box are obtained from the hand bounding box width and height heat map and the hand bounding box offset heat map, respectively. The hand bounding box is determined based on the first width value, the first height value, and the first offset value.

6. The method according to claim 1, characterized in that, The hand bounding box information includes the center position coordinates of the hand bounding box, the second width value, and the second height value; The step of suppressing the heatmap of the target hand bounding box corresponding to the non-existent wrist based on the hand bounding box information, and determining the hand bounding box based on the suppression operation result, includes: Obtain the minimum value between the second height value and the second width value, and determine the target area in the target hand bounding box heat map based on the minimum value and the center position coordinates; The probability values ​​corresponding to each location coordinate within the target area are weighted to obtain the suppression operation result; Obtain the second maximum value of the probability of a hand bounding box existing in the suppression operation result, the second position coordinates corresponding to the second maximum value, and the fourth confidence level; If the fourth confidence level is greater than or equal to the preset confidence threshold, then based on the second position coordinates, the third width value, the third height value, and the second offset value of the corresponding hand bounding box are obtained from the hand bounding box width and height heat map and the hand bounding box offset heat map, respectively. The bounding box of the hand corresponding to the non-existent wrist is determined based on the third width value, the third height value, and the second offset value.

7. The method according to claim 1, characterized in that, Before inputting the human body region image into the limb capture network, the method further includes: Obtain a training set and train the limb capture network to be trained based on the training set to obtain the limb capture network; The training set includes human body region image samples and labeled hand bounding boxes corresponding to the human body region image samples.

8. The method according to claim 7, characterized in that, The step of training the limb capture network to be trained based on the training set to obtain the limb capture network includes: The human body region image samples are input into the limb capture network to be trained to obtain a heat map set of predicted hand bounding boxes; The loss function corresponding to the limb capture network to be trained is determined based on the predicted hand bounding box heatmap and the labeled hand bounding box. The parameters of the limb capture network to be trained are optimized according to the loss function to obtain the limb capture network.

9. The method according to claim 8, characterized in that, The predicted hand bounding box heatmap set includes a predicted hand bounding box heatmap, a predicted hand bounding box width and height heatmap, and a predicted hand bounding box offset heatmap. The step of determining the loss function of the limb capture network based on the predicted hand bounding box heatmap and the labeled hand bounding boxes includes: The first loss function is determined based on the predicted hand bounding box heatmap and the coordinates of the center position of the marked hand bounding box; The second loss function is determined based on the predicted hand bounding box width and height heatmap and the fourth width and fourth height values ​​of the hand bounding box. The third loss function is determined based on the predicted hand bounding box offset heatmap and the coordinates of the center position of the marked hand bounding box; The loss function is constructed based on the first loss function, the second loss function, and the third loss function.

10. The method according to claim 9, characterized in that, The step of determining the first loss function based on the predicted hand bounding box heatmap and the coordinates of the center position of the labeled hand bounding box includes: The supervision function for the predicted hand bounding box heatmap is determined based on the center coordinates of the labeled hand bounding box. Obtain the first difference between the predicted hand bounding box heatmap and the supervision function; The first loss function is determined based on the first difference.

11. The method according to claim 9, characterized in that, The step of determining the second loss function based on the predicted hand bounding box width and height heatmap and the labeled width and height of the hand bounding box includes: Obtain the fifth width and fifth height values ​​of the predicted hand bounding box heatmap at the center position of the marked hand bounding box; Calculate the second difference between the fifth width value and the fourth width value, and calculate the third difference between the fifth height value and the fourth height value; The second loss function is determined based on the second difference and the third difference.

12. The method according to claim 9, characterized in that, The step of determining the third loss function based on the predicted hand bounding box offset heatmap and the coordinates of the center position of the labeled hand bounding box includes: The coordinates of the center position of the marked hand bounding box are rounded to obtain the rounded result. Obtain the third bias value of the predicted hand bounding box bias heatmap at the rounded result; Calculate the fourth difference between the center position coordinates of the marked hand bounding box and the rounded result; Calculate the fifth difference between the third bias value and the fourth difference; The third loss function is determined based on the fifth difference.

13. The method according to claim 1, characterized in that, After inputting the human body region image into the limb capture network to obtain the hand bounding box and limb motion, the method further includes: Obtain the third confidence level of the left bounding box and the fourth confidence level of the right bounding box respectively; If both the third confidence level and the fourth confidence level are greater than or equal to the preset confidence threshold, then the intersection-union ratio of the left-hand bounding box and the right-hand bounding box is obtained; If the cross-union ratio is greater than or equal to a preset cross-union ratio threshold, then the two-hand bounding boxes are obtained based on the left-hand bounding box and the right-hand bounding box.

14. The method according to claim 1, characterized in that, The step of obtaining hand movements based on the hand bounding box includes: The human body region image is cropped using the hand bounding box to obtain a hand region image; The hand region image is input into a hand capture network, which extracts features from the hand region image to obtain the hand movement.

15. The method according to claim 1, characterized in that, The step of obtaining the human body region image from the image to be identified includes: The image to be identified is input into a human detection network, and human detection is performed on the image to be identified by the human detection network to obtain a human bounding box; The human body bounding box is used to crop the image to be identified to obtain the human body region image.

16. A motion capture device, characterized in that, include: The human body region image acquisition module is configured to acquire human body region images from the image to be identified; The hand bounding box acquisition module is configured to input the human body region image into the limb capture network to obtain the hand bounding box and limb movement. The hand bounding box is determined based on the hand bounding box heat map set, which includes the hand bounding box heat map, the hand bounding box width and height heat map, and the hand bounding box offset heat map. The hand motion acquisition module is configured to acquire hand motion based on the hand bounding box; The human motion acquisition module is configured to fuse the hand movements and the limb movements to obtain human motion. The hand bounding box acquisition module is further configured to acquire a two-dimensional key point heatmap of the human body region image, and to acquire a first confidence level of the wrist key point from the two-dimensional key point heatmap. The presence of a wrist is determined based on the first confidence level, and the bounding box of the hand is determined based on the determination result; Determining the hand bounding box based on the judgment result includes: When it is determined that only the left wrist or the right wrist exists, extract the bounding box information of the hand corresponding to the existing wrist; Based on the hand bounding box information, a suppression operation is performed on the heatmap of the target hand bounding box corresponding to the non-existent wrist, and the hand bounding box is determined based on the suppression operation result.

17. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the motion capture method as described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the motion capture method as described in any one of claims 1 to 15.

Images