A gesture image processing method, storage medium and device

Abstract

This specification provides a gesture image processing method, storage medium, and device. In this method, image features of a target gesture image are acquired and fed into a keypoint detection network for keypoint detection. The gesture keypoint information output by the network is combined with the image features and fed into a 3D reconstruction network for 3D reconstruction, resulting in a 3D hand network. These two networks are jointly trained, and the resulting output is used to drive the target 3D model. This method reduces processing overhead by jointly training and inferring the deep neural network performing hand keypoint detection and the deep neural network performing gesture 3D reconstruction, thereby meeting real-time processing requirements.

Description

Technical Field

[0001] This specification relates to the field of image processing technology, and in particular to a gesture image processing method, storage medium, and device. Background Art

[0002] In AR (Augmented Reality) or VR (Virtual Reality) applications, it's often necessary to drive a 3D model based on gestures in captured images. However, the time-consuming 3D model-driving process in related technologies can't meet the demands of some applications, such as live streaming, which requires high real-time performance. Summary of the Invention

[0003] According to a first aspect of an embodiment of this specification, a gesture image processing method is provided, including: obtaining image features of a target gesture image; inputting the image features into a key point detection network for key point detection to obtain gesture key point information; inputting the image features and gesture key point information into a three-dimensional reconstruction network for three-dimensional reconstruction to obtain a three-dimensional hand mesh; the key point detection network and the three-dimensional reconstruction network are obtained by joint training based on multiple gesture image samples; and driving a target three-dimensional model based on the gesture key point information and the three-dimensional hand mesh.

[0004] In some embodiments, the above-mentioned image features include a first feature output by the Mth layer of the feature extraction network and a second feature output by the Nth layer of the feature extraction network, where M is less than N; the image feature input into the three-dimensional reconstruction network is the first feature, and the image feature input into the key point detection network is the second feature.

[0005] In some embodiments, the above-mentioned driving of the target three-dimensional model based on the gesture key point information and the three-dimensional hand mesh includes: inputting the image features into a gesture segmentation network to obtain a gesture mask; and driving the target three-dimensional model based on the gesture key point information, the three-dimensional hand mesh and the gesture mask.

[0006] In some embodiments, the above-mentioned image features include a first feature output by the Mth layer of the feature extraction network and a second feature output by the Nth layer of the feature extraction network, where M is less than N; the image feature input to the three-dimensional reconstruction network is the first feature, and the image feature input to the key point detection network and the image feature input to the gesture segmentation network are both second features.

[0007] In some embodiments, the above-mentioned three-dimensional reconstruction network includes a first sub-network and a second sub-network; the inputting the image features and gesture key point information into the three-dimensional reconstruction network for three-dimensional reconstruction to obtain a three-dimensional hand mesh includes: inputting the first features and the gesture key point information into the first sub-network to generate reconstructed mesh features, and constructing an initial hand mesh based on the reconstructed mesh features; inputting the initial hand mesh, the first features and the reconstructed mesh features into the second sub-network to generate residual information, and adjusting the initial hand mesh based on the residual information to obtain a three-dimensional hand mesh.

[0008] In some embodiments, the residual information is obtained by inputting a combined feature into the second sub-network; the combined feature is obtained by converting the initial hand mesh, the first feature, and the reconstructed mesh feature into a specified dimension and then superimposing them.

[0009] In some embodiments, the reconstructed mesh features input to the second sub-network include the third features output by the S-th layer of the first sub-network and the fourth features output by the T-th layer of the first sub-network, where S is less than T.

[0010] According to a second aspect of an embodiment of this specification, a gesture image processing method is provided, including: obtaining a gesture image and a corresponding three-dimensional hand mesh, the gesture image being marked with gesture key points; obtaining image features of the gesture image, inputting the image features into a key point detection network to obtain predicted key points, and obtaining a first difference between the predicted key points and the marked gesture key points; inputting the image features and the predicted key points into a three-dimensional reconstruction network to obtain a predicted hand mesh, and obtaining a second difference between the predicted hand mesh and the three-dimensional hand mesh; constructing a loss function based on the first difference and the second difference, and using the loss function to jointly train the key point detection network and the three-dimensional reconstruction network.

[0011] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, any one of the methods in the embodiments of the specification is implemented.

[0012] According to a fourth aspect of the embodiments of this specification, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the methods in the embodiments of the specification is implemented.

[0013] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0014] In embodiments of this specification, a gesture image processing method, storage medium, and device are disclosed. This method obtains image features of a target gesture image and inputs them into a keypoint detection network for keypoint detection. The gesture keypoint information output by the network is combined with the image features and input into a 3D reconstruction network for 3D reconstruction, resulting in a 3D hand network. These two networks are jointly trained, and the output is used to drive the target 3D model. This method reduces processing overhead by jointly training and inferring the deep neural network performing the hand keypoint detection task and the deep neural network performing the gesture 3D reconstruction task, thereby meeting real-time processing requirements.

[0015] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0017] Figure 1 is a flowchart of a gesture image processing method according to an exemplary embodiment of this specification;

[0018] Figure 2A is a schematic diagram of a network structure of a Graph-CNN used in a second sub-network according to an exemplary embodiment of this specification;

[0019] Figure 2B is a schematic diagram illustrating a result of three-dimensional reconstruction of an input image according to an exemplary embodiment of this specification;

[0020] Figure 3 is a flowchart of another gesture image processing method according to an exemplary embodiment of this specification;

[0021] Figure 4 is a schematic diagram of a network framework for gesture image processing according to an exemplary embodiment of this specification;

[0022] Figure 5 This is a hardware structure diagram of a computer device where a gesture image processing device is located according to an exemplary embodiment of this specification;

[0023] Figure 6 is a block diagram of a gesture image processing device according to an exemplary embodiment of the present specification;

[0024] Figure 7 is a block diagram of another gesture image processing device according to an exemplary embodiment of this specification. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this specification. Rather, they are merely examples of apparatus and methods consistent with certain aspects of this specification, as detailed in the appended claims.

[0026] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. As used in this specification and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0027] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information without departing from the scope of this specification. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0028] In AR (Augmented Reality) or VR (Virtual Reality) applications, it's often necessary to drive a 3D model based on gestures in captured images. However, the time-consuming 3D model-driving process in related technologies can't meet the demands of some applications, such as live streaming, which requires high real-time performance.

[0029] Based on this, the embodiment of this specification provides a gesture image processing method to solve the above problem. The embodiment of this specification is described in detail below.

[0030] like Figure 1 As shown, Figure 1 FIG. 1 is a flowchart of a gesture image processing method according to an exemplary embodiment of the present specification, wherein the method includes:

[0031] Step 101: Acquire image features of a target gesture image;

[0032] The method of this embodiment can be applied to application software that needs to perform tasks such as hand key point detection and gesture 3D reconstruction, such as applications that display AR special effects, virtual live broadcast applications, etc. The application software can be installed on portable terminals such as smartphones and tablets, and can also be installed on terminals such as interactive tablets and desktop computers.

[0033] The target gesture image mentioned in this step refers to an image showing the target gesture. This image can be captured in real time by a camera or uploaded by the user. The target gesture can refer to the shape and posture of the hand presented in the image. The target gesture can be any gesture, such as a fist, a thumbs-up, an "OK" sign, etc.

[0034] The image features of the target gesture image can be "non-image" descriptions of the characteristics and / or properties of the target gesture image. This "non-image" description can be a feature value or a feature vector. The image features can be obtained using traditional feature extraction algorithms, such as Scale-Invariant Feature Transform (SIFT) and Histogram of Oriented Gradients (HOG). In some examples, the image features can be obtained by inputting the target gesture image into a feature extraction network. The feature extraction network can be a deep neural network, optionally a ResNet (Residual Network). ResNet is a family of convolutional neural networks (CNNs), including ResNet34, ResNet50, and ResNet101. The residual blocks within ResNet use skip connections to alleviate the vanishing gradient problem associated with increasing the depth of deep neural networks. Compared to traditional feature extraction algorithms, this feature extraction network has stronger expressive power, and therefore the image features obtained using this feature extraction network contain richer information.

[0035] Step 102: Input the image features into a key point detection network to perform key point detection to obtain gesture key point information;

[0036] Gesture keypoints are nodes that locate the hand. The human hand generally has 21 gesture keypoints, corresponding to 21 major skeletal points, including the fingertips, phalangeal joints, and the wrist. Gesture keypoint detection is a crucial component of gesture recognition and a key step in practical applications such as human-computer interaction, virtual reality, and augmented reality.

[0037] The gesture key point information of the target gesture image may include features representing the position coordinates of each gesture key point in the target gesture image. The position coordinates may be the three-dimensional coordinates of the gesture key points in camera space. In this embodiment, the gesture key point information is obtained by inputting image features into a key point detection network. The key point detection network can perform network inference on the image features to generate multiple heat maps, each heat map corresponding to a gesture key point. The heat map corresponding to each gesture key point represents the distribution of the position of the gesture key point in the heat map. Each pixel on the heat map can be associated with a probability value, which represents the probability of the gesture key point being at that location. Therefore, the position coordinates of each gesture key point can be obtained through these heat maps. The key point detection network is a deep neural network. In some examples, the key point detection network can adopt a lightweight deep neural network, such as Lite-HRNet (Lite High-Resolution Network) or SqueezeNet (SqueezeNetwork). Taking Lite-HRNet as an example, this network applies the efficient Shuffle block to the traditional HRNet and introduces a lightweight unit called conditional channel weighting to replace the 1×1 convolution in the Shuffle block to reduce the amount of computation. Moreover, this network connects high-resolution to low-resolution subnetworks in parallel. Throughout the process, information is repeatedly exchanged on parallel multi-resolution subnetworks to perform multi-scale fusion, making the final predicted heat map more accurate and spatially precise.

[0038] After obtaining the heat map of each gesture key point, the position coordinates of each gesture key point can be obtained by calculating the position coordinates of the maximum value in the heat map. In an optional embodiment, the position coordinates of each gesture key point can be obtained by calculating the heat map using the integration method (soft-argmax), such as the position coordinates of a gesture key point (J x , J y , J z ) can be based on the following formula, from the heat map F of the gesture key point j Calculation yields:

[0039] J x =softargmax(Conv 1D (avg x (F j )))

[0040] J y =softargmax(Conv 1D (avg y (F j )))

[0041] J z =Conv 1D (Φ(avg x,y (F j )))

[0042] In the above formula, x and y represent the x and y spatial coordinates respectively; z represents the distance from the camera to the hand; Conv 1D Represents a 1D convolution; Φ represents a fully connected layer, which can be a 1D BN (Batch Normalization) function; avg x Indicates that the x-axis is averaged. Accordingly, avg y Indicates the calculation of the average of the y-axis, avg x,y Indicates that the average of the x-axis and y-axis is calculated simultaneously. By processing the heat map of each gesture key point in turn using the above formula, the position coordinates of each gesture key point can be obtained.

[0043] Of course, in other embodiments, the position coordinates of each gesture key may be obtained from the heat map by other methods, such as NMS (Non-Maximum Suppression), and this specification does not limit this.

[0044] Step 103: Input the image features and gesture key point information into a 3D reconstruction network for 3D reconstruction to obtain a 3D hand mesh; the key point detection network and the 3D reconstruction network are jointly trained based on multiple gesture image samples;

[0045] The three-dimensional hand mesh can be a mesh model of the hand. The three-dimensional reconstruction of the hand is also a key step in practical applications such as human-computer interaction, virtual reality, and augmented reality. For example, in a virtual environment, the reconstructed mesh model can be used to realize interactive tasks such as grasping and releasing other virtual objects.

[0046] The three-dimensional hand mesh of the target gesture image can be a Mesh model corresponding to the target gesture presented in the target gesture image, and can be composed of the coordinates of multiple vertices. In this embodiment, the three-dimensional hand mesh is obtained by inputting image features and gesture key point information into a three-dimensional reconstruction network. The three-dimensional reconstruction network can embed image features into each vertex of a preset mesh template, so that each vertex has an initial coordinate, and then adjust it through the gesture key point information to obtain the final coordinates of each vertex. Optionally, the network structure adopted by the three-dimensional reconstruction network can also be ResNet. Of course, in other embodiments, the three-dimensional reconstruction network can adopt other types of deep neural networks. The specific calculation process of the deep neural network can be found in the description of the relevant technology, and this specification will not go into details here.

[0047] In this embodiment, the three-dimensional hand mesh and the key point detection network mentioned in step 102 are obtained by joint training based on multiple gesture image samples. That is, multiple gesture image samples can be used as a training set, and the training set can include multiple gesture images and a three-dimensional hand mesh corresponding to each gesture image in the multiple gesture images, wherein each gesture image is marked with a gesture key point. In this way, in the training stage, a loss function can be constructed by combining the first difference between the marked gesture key point and the predicted key point predicted by the key point detection network, and the second difference between each three-dimensional hand mesh and the predicted hand mesh predicted by the three-dimensional reconstruction network, and the loss function is used to jointly train the two networks. The specific training process can be found in Figure 3 Related description of the embodiment shown.

[0048] In addition to joint training, in this embodiment, the output of the keypoint detection network can be used as input to the 3D reconstruction network, enabling joint reasoning. In other words, this embodiment effectively performs joint training and reasoning for both hand keypoint detection and gesture 3D reconstruction. Compared to separate approaches in related technologies, this reduces processing effort and improves task efficiency.

[0049] In addition, in some examples, the image features may include a first feature output by the Mth layer of the feature extraction network and a second feature output by the Nth layer of the feature extraction network, where M is less than N; the image features input to the 3D reconstruction network are the first features, and the image features input to the key point detection network are the second features. The feature extraction network may include multiple computational layers, with different computational layers having different semantic levels. For example, the features output by the second layer contain less image detail information than the features output by the first layer, but the semantic expression is more abstract. Therefore, this embodiment uses the output features of the appropriate computational layer in the feature extraction network as input based on the differences in the tasks performed by the two networks. The first feature containing more image detail information of the target gesture image is used as the input of the 3D reconstruction network that performs 3D gesture reconstruction, and the second feature, which has a more abstract semantic expression of the target gesture image, is used as the input of the key point detection network that performs the hand key point detection task. This improves the processing effect of the two networks.

[0050] Optionally, the first feature may be the output result of the first layer of the feature extraction network, and the second feature may be the output result of the last layer of the feature extraction network. Taking the feature extraction network as ResNet50 as an example, the feature extraction network includes multiple convolutional layers, each convolutional layer outputs a feature map, and the spatial dimension of the feature map output by the first layer is larger than the spatial dimension of the feature map output by the last layer. Therefore, the first feature represents local information, and the second feature represents global information. The task that the key point detection network needs to perform is to automatically discover semantically meaningful and fixed-position gesture key points. Therefore, using the first feature as the input of the key point detection network can improve the accuracy of gesture key point detection; and through experiments, it is found that when performing three-dimensional reconstruction based on gesture key point information, supplementing the local information of the target gesture image as the input of the three-dimensional reconstruction network can obtain better results. It should be noted that, in some other examples, M and N can be a range of values, where the maximum value in the range of values ​​corresponding to M is smaller than the minimum value in the range of values ​​corresponding to N. For example, a feature extraction network is a deep neural network comprising 18 convolutional layers. The first feature can be a combination of features output by the 1st to 3rd convolutional layers, and the second feature can be a combination of features output by the 15th to 18th convolutional layers.

[0051] Step 104: Drive the target three-dimensional model based on the gesture key point information and the three-dimensional hand mesh.

[0052] After obtaining the gesture key point information and three-dimensional hand mesh corresponding to the target gesture image, the target three-dimensional model can be driven based on the gesture key point information and three-dimensional hand mesh to realize the corresponding business. The business here can be AR special effects processing, VR special effects processing, gesture drive in the concept of "metaverse", etc. Depending on the business, the target three-dimensional model and the corresponding driving method may also be different. For example, in AR special effects processing, the target three-dimensional model can be a three-dimensional model in the AR special effects material; in virtual live broadcast, the target three-dimensional model can be the hand model of the virtual anchor.

[0053] Taking the target three-dimensional model as a virtual water cup in the AR scene as an example, the virtual water cup is a three-dimensional model of a water cup image. The terminal can obtain the target gesture image presenting the user's current gesture, and obtain the gesture key point information and the three-dimensional hand grid based on the solution of this embodiment. Then, based on the gesture key point information and the distance between the three-dimensional hand grid and the virtual water cup, it is judged whether the user's current gesture meets the requirements for grabbing the virtual water cup. If so, the virtual water cup is controlled to move with the movement of the user's hand; otherwise, the virtual water cup is controlled to stay in its original position. In this way, interaction with the virtual object in the AR scene is achieved.

[0054] This embodiment's solution acquires image features from the target gesture image and feeds them into a keypoint detection network for keypoint detection. The output, combined with the image features, is then fed into a 3D reconstruction network for 3D reconstruction, resulting in a 3D hand network. These two networks are jointly trained, and their outputs are used to drive the target 3D model. This approach reduces processing overhead by jointly training and inferring the deep neural network performing hand keypoint detection and the deep neural network performing 3D gesture reconstruction, enabling real-time processing of both tasks.

[0055] In some business scenarios, in addition to gesture key points and a three-dimensional hand network, a gesture mask is also required to achieve accurate driving of the target three-dimensional model. A mask, also known as a mask, is an image or graphic used to block the image to be processed to control the area or processing process of the image. The gesture mask is usually obtained by performing gesture segmentation on the collected gesture image. In related technologies, the task of gesture segmentation is generally performed by an independent deep learning model. In some embodiments of this specification, the task of gesture segmentation can be jointly trained and jointly inferred with the two tasks of hand key point detection and gesture three-dimensional reconstruction. Specifically, the gesture mask of the target gesture image can be obtained by inputting the image features obtained in step 101 into the gesture segmentation network. The gesture segmentation network and the key point detection network and the three-dimensional reconstruction network mentioned above can be jointly trained to drive the target three-dimensional model based on the gesture key point features, the three-dimensional hand mesh and the gesture mask. In this case, the training set for joint training can include multiple gesture images and the three-dimensional hand mesh corresponding to each of the multiple gesture images, wherein each gesture image is marked with gesture key points and a gesture mask. Optionally, the gesture segmentation network can adopt a lightweight network structure such as FCN (Fully Convolutional Network), which can recover the category to which each pixel belongs from the image features through deconvolution, thus obtaining a gesture mask. Accordingly, when the image features include a first feature and a second feature, since the task that the gesture segmentation network needs to perform is to automatically complete the classification of pixels with different semantic meanings, the gesture mask can be obtained by inputting the second feature into the gesture segmentation network. It should be noted that in some examples, the gesture mask output by the gesture segmentation network can also be combined with image features as the input of the key point detection network to enhance the output effect of the key point detection network.

[0056] For the task of 3D hand gesture reconstruction, in some examples, the 3D reconstruction network may include a first subnetwork and a second subnetwork. Step 103 may include: inputting the first feature and gesture key point information into the first subnetwork to generate reconstructed mesh features, and constructing an initial hand mesh based on the reconstructed mesh features; inputting the initial hand mesh, the first feature, and the reconstructed mesh features into the second subnetwork to generate residual information, and adjusting the initial hand mesh based on the residual information to obtain a 3D hand mesh. The first subnetwork may include a Mesh encoder and a Mesh decoder. Optionally, the Mesh encoder may be a ResNet encoder. The Mesh encoder may use the first feature and gesture key point information of the target gesture image as input data. The Mesh encoder may encode the input data to obtain reconstructed mesh features. The reconstructed mesh features may represent low-dimensional feature values ​​of the input data. After reconstruction by the Mesh decoder, a "rough" reconstructed Mesh, i.e., the initial hand mesh, may be obtained. The initial hand mesh may include features of multiple vertices, and the so-called "roughness" may mean that the initial hand mesh is not completely aligned with the hand in the target gesture image, such as the features of some vertices are not accurate enough or are missing. Therefore, the initial hand mesh can be refined by a second sub-network. The second sub-network can use the initial hand mesh, the first features, and the reconstructed mesh features to capture more local information to generate residual information. This residual information is used to adjust the vertex state of the initial hand mesh, such as adding new vertices, adjusting the coordinates of the original vertices, etc., so that the adjusted target hand mesh is more accurately aligned with the hand in the target gesture image.

[0057] In some examples, the second sub-network can be a Graph-CNN (Graph Convolutional Network). The reconstructed mesh is essentially a graph, and each vertex of the reconstructed mesh is a node of the graph. Graph-CNN can effectively extract spatial features for learning based on the graph data structure, making it suitable for refining the initial hand mesh. Specifically, the graph convolution layer of the second sub-network can adopt the convolution layer of the GIN (Graph Isomorphism Network). The calculation process of the GIN convolution layer can be as follows:

[0058]

[0059] In the above formula, x i represents the characteristics of the i-th node; x i ′represents the updated features of the i-th node; N(i) represents the set of neighboring nodes of the i-th node; and MLPs represents mapping through multiple MLPs (Multi-Layer Perceptrons). Through this computational process, the second sub-network can obtain more information about the node's neighbors and use it to update the state of each node.

[0060] like Figure 2A As shown, Figure 2A : This is a schematic diagram of the Graph-CNN network structure used by the second sub-network according to an exemplary embodiment of this specification. The input Mesh feature of the second sub-network can be called a combined feature. The combined feature is composed of the initial hand mesh, the first feature, and the reconstructed mesh feature. Optionally, the combined feature can be obtained by converting the initial hand mesh, the first feature, and the reconstructed mesh feature to a specified dimension and then superimposing them. That is, when applying the second sub-network, the initial hand mesh, the first feature, and the reconstructed mesh feature can first be adjusted to the same dimension, and then simply superimposed to form a combined feature, which is then input into the second sub-network to obtain residual information. In particular, when constructing the combined feature, the introduction of the first feature containing more image details can supplement local information for the second sub-network to enhance its output effect. In some examples, the reconstructed mesh feature used in constructing the combined feature can include the third feature output by the Sth layer of the first sub-network and the fourth feature output by the Tth layer of the first sub-network. S is less than T. Correspondingly, the spatial dimension of the third feature is larger than the spatial dimension of the fourth feature. Experiments have found that this can make the output result of the second sub-network more accurate.

[0061] Depend on Figure 2AAs can be seen, the second sub-network includes several homogeneous residual blocks, namely GINResBlock. Each homogeneous residual block contains two homogeneous convolution blocks, namely GINConvBlock, and a residual value. Each homogeneous convolution block consists of a GIN convolution layer, a one-dimensional batch normalization function, and a ReLU (Rectified Linear Unit) activation function. The two homogeneous convolution layers in each homogeneous residual block can each update the node state based on the above calculation process. The difference between the calculation results of these two homogeneous convolution layers is the residual value of the homogeneous residual block. The input data of one homogeneous residual block and the residual value of the homogeneous residual block are superimposed and serve as the input data of the next homogeneous residual block. The last homogeneous convolution block can adjust the feature output so that the residual information ΔM ultimately output by the second sub-network represents the offset of each node. In this way, this offset is added to the features of each vertex in the initial hand mesh output by the first sub-network. The result of this addition constitutes the target hand mesh. This solution uses the coarse-to-fine approach to perform gesture 3D reconstruction. First, a rough reconstruction result is predicted. Then, based on this reconstruction result, multi-dimensional features (target gesture image features, key point detection network features, and first sub-network features) are integrated and input into Graph-CNN. Graph-CNN then performs graph convolution operations from coarse to fine to obtain the final refined gesture 3D reconstruction result. Figure 2B As shown, Figure 2B This is a schematic diagram of the results of 3D reconstruction of some input images based on this scheme. Figure 2B It can be seen that based on this scheme, fine-grained gesture 3D reconstruction results can be obtained, which are highly consistent with the gestures in the input image and have a strong sense of realism.

[0062] It should be noted that in other embodiments, other deep neural networks, such as GGNN (Gated Graph Neural Network) can also be used as the second sub-network, and this specification does not limit this.

[0063] For the several networks involved in the above methods, this manual also provides corresponding training methods, such as Figure 3 As stated, Figure 3 FIG. 1 is a flowchart of a gesture image processing method according to an exemplary embodiment of the present specification, wherein the method includes:

[0064] Step 301: Acquire a gesture image and a corresponding three-dimensional hand mesh, wherein the gesture image is marked with gesture key points;

[0065] The gesture images and corresponding three-dimensional hand meshes obtained in this step can constitute a training set, which may include multiple gesture images presenting different gestures, and a three-dimensional hand mesh generated for each gesture image, wherein the gesture key points marked in each gesture image can be three-dimensional key points corresponding to the hand in the gesture image, and the coordinates of these gesture key points can be added to the information of the gesture image in the form of labels.

[0066] Step 302: Obtain image features of the gesture image, input the image features into a key point detection network to obtain predicted key points, and obtain a first difference between the predicted key points and the marked gesture key points;

[0067] The first difference represents the output error of the key point detection network. In some examples, the first difference can be calculated using L1-Loss, as shown in the following formula:

[0068]

[0069] In the above formula, L joint represents the first difference; J i represents the i-th predicted key point; In other embodiments, other methods such as L2-loss and Wing-Loss can also be used to calculate the output error of the key point detection network.

[0070] Step 303: Input the image features and the predicted key points into a 3D reconstruction network to obtain a predicted hand mesh, and acquire a second difference between the predicted hand mesh and the 3D hand mesh;

[0071] The second difference represents the output error of the 3D reconstruction network, and the second difference can also be calculated using L1-Loss. When the 3D reconstruction network includes a first sub-network and a second sub-network, the second difference can be calculated based on the following formula:

[0072]

[0073] In the above formula, L mesh represents the second difference; represents the i-th node in the 3D hand mesh; represents the i-th node of the initial hand mesh output by the first sub-network; represents the ith node of the predicted hand mesh finally output by the 3D reconstruction network. Similarly, in other embodiments, other methods may be used to calculate the output error of the 3D reconstruction network.

[0074] Step 304: construct a loss function based on the first difference and the second difference, and use the loss function to jointly train the key point detection network and the 3D reconstruction network.

[0075] In this embodiment, the key point detection network and the 3D reconstruction network are jointly trained in an end-to-end manner. The first difference and the second difference can be constructed into a total difference. For example, the total difference is:

[0076] L=λ j L joint +λ m L mesh

[0077] In the above formula, L represents the total difference; λ j and λ m The weights representing the first and second differences, respectively, can be set based on the needs of the specific scenario. After obtaining the total difference, a loss function can be constructed based on the total difference. This loss function is then used to update the keypoint detection network and the 3D reconstruction network, with multiple iterations until the loss function is minimized. In this way, the trained keypoint detection network and 3D reconstruction network can be used to jointly complete the tasks of hand keypoint detection and hand gesture 3D reconstruction, with higher processing efficiency.

[0078] In addition, the solution of this embodiment can also jointly train the gesture segmentation network used to perform the gesture mask acquisition task. Specifically, the gesture images in the training set can be marked with gesture masks, that is, a classification label is set for each pixel in the gesture image, and the output error of the gesture segmentation network is recorded as the third difference. In some examples, the third difference can be calculated using a cross entropy loss function. Specifically, the third difference can be calculated based on the following formula:

[0079]

[0080] In the above formula, L seg represents the third difference; H and W represent the height and width of the gesture mask respectively; y j represents the label of the j-th pixel, p j Represents the probability corresponding to the jth pixel in the prediction result of the gesture segmentation network. The calculated third difference can be combined with the first difference and the second difference to construct a loss function, such as the total difference is:

[0081] L=λ j L joint +λ m L mesh +λ s L seg

[0082] In the above formula, λ s represents the weight of the third difference, and the meanings of the remaining parameters are consistent with the above. In an optional embodiment, λ j Can be 1, λ m Can be 1, λ s It can be 10.

[0083] In order to explain the gesture image processing method of this specification in more detail, a specific embodiment is introduced below:

[0084] This embodiment relates to an application software for AR special effects processing, which can display an AR scene with some virtual objects, such as virtual toys, virtual water cups, etc. At the same time, the application software can capture the gesture image of the target gesture displayed by the user's hand through a camera, and perform tasks such as gesture segmentation, hand key point detection, and gesture 3D reconstruction on the gesture image, and then drive the virtual object based on the results of each task to achieve interaction with the virtual object.

[0085] In related technologies, application software generally establishes a deep learning model for each task, and then processes the gesture images based on each deep learning model. The reasoning takes a long time, and generally only 10 to 15 frames of images can be processed per second, which does not meet the real-time processing requirements. The displayed results are not realistic enough for users.

[0086] When the gesture image processing method of this specification is applied to the application software, real-time processing can be achieved, improving the user experience. Figure 4 The network framework shown, wherein the input image 41 is a gesture image captured in real time by the camera during one processing of the application software.

[0087] Image Encoder 42 includes a feature extraction network that uses a ResNet structure to extract image features of the input image 41. The image features output by the first convolutional layer of the feature extraction network are the first features F l , the image feature output by the last convolutional layer of the feature extraction network is the second feature F i , these two image features are input to the three branches as the basic features of the three branches.

[0088] The three branches are gesture segmentation branch, hand key point detection branch and gesture 3D reconstruction branch. Among them, the gesture segmentation branch uses Segment Decoder43 to input the second feature F iPerform gesture segmentation to obtain a gesture mask 44, which is a binary classification mask with the same resolution as the input image 41. The Segment Decoder 43 includes a gesture segmentation network, which uses a lightweight network structure similar to FCN and uses a cross entropy loss function to calculate the error L seg ;

[0089] In the hand key point detection branch, Joint Decoder45 is used to input the second feature F i Perform key point detection to obtain a 21-channel heat map. Process the heat map with the soft-argmax function to obtain 21 gesture key points 46 with three-dimensional coordinates. The characteristics of the gesture key points 46 are F j The Joint Decoder45 includes a key point detection network that uses a lightweight HRNet structure and uses L1-Loss to calculate the error L joint ;

[0090] The gesture 3D reconstruction branch includes the Coarse stage and the Refinement stage. In the Coarse stage, the gesture 3D reconstruction branch uses the first feature F of the input of Mesh Encoder47 l And the feature of gesture key point 46 is F j Perform 3D reconstruction to obtain reconstructed mesh features, and then use Mesh Decoder 48 to process the reconstructed mesh features to obtain an initial hand mesh 49. The resolution of the initial hand mesh 49 is one-fourth of the resolution of the input image 41. The features of the initial hand mesh 49 are F c The Mesh Encoder 47 includes a first sub-network, which uses a ResNet structure. The reconstructed mesh feature output by the first convolution layer of the first sub-network is the third feature F m , the reconstructed grid feature output by the last convolutional layer of the first sub-network is the fourth feature F g ; In the Refinement stage, the first feature F l 、The third feature F m 、Fourth feature F g And the initial hand mesh 49 is characterized by F cAfter adjusting to a unified dimension, the combined features are superimposed and input into Graph-CNN (denoted as 50 in the figure). The graph convolution layer of Graph-CNN uses the convolution layer of GIN to process the combined features and obtain residual information ΔM. The residual information ΔM is used to adjust the initial hand grid 49 to obtain the target hand grid 51. The number of nodes in the target hand grid 51 is more than 6 times the number of nodes in the initial hand grid 49, which is more refined. The error L in the whole process of the gesture 3D reconstruction branch is mesh Use L1-Loss calculation;

[0091] The entire network is trained simultaneously through End-to-End, and the total loss is:

[0092] L=λ j L joint +λ m L mes h+λ s L seg

[0093] Among them, λ j =1,λ m =1,λ s =10; the training dataset uses the open source dataset GT, which includes a large number of gesture images, as well as the segmentation masks, 3D key points and / or 3D gesture meshes corresponding to each gesture image; the dataset includes real data and data synthesized using simulation software.

[0094] Experiments have shown that the framework takes an average of 25ms per frame to infer gesture image sequences captured by a consumer-grade monocular camera. This means it can process 40 frames per second, meeting the real-time processing requirement of 25 FPS (Frames Per Second). Furthermore, the framework achieves high accuracy in the three tasks of gesture segmentation, hand keypoint detection, and 3D gesture reconstruction, resulting in more realistic AR effects.

[0095] Corresponding to the aforementioned method embodiments, this specification also provides embodiments of a gesture image processing device and a terminal to which the device is applied.

[0096] The embodiments of the gesture image processing device of this specification can be applied to computer devices, such as servers or terminal devices. The device embodiments can be implemented through software, hardware, or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the file processing in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, such as Figure 5The figure shows a hardware structure diagram of the computer device where the gesture image processing device of the embodiment of this specification is located. Figure 5 In addition to the processor 510, memory 530, network interface 520, and non-volatile memory 540 shown, the server or electronic device where the device 531 is located in the embodiment may also include other hardware according to the actual function of the computer device, which will not be described in detail.

[0097] Accordingly, an embodiment of this specification further provides a computer storage medium, wherein the storage medium stores a program, and when the program is executed by a processor, the method in any of the above embodiments is implemented.

[0098] The embodiments of this specification may take the form of a computer program product implemented on one or more storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0099] like Figure 6 As shown, Figure 6 : is a block diagram of a gesture image processing device according to an exemplary embodiment of the present specification, the device comprising:

[0100] A feature acquisition module 61 is used to acquire image features of a target gesture image;

[0101] A first input module 62 is used to input the image features into a key point detection network to perform key point detection and obtain gesture key point information;

[0102] A second input module 63 is configured to input the image features and gesture key point information into a 3D reconstruction network for 3D reconstruction to obtain a 3D hand mesh; the key point detection network and the 3D reconstruction network are jointly trained based on multiple gesture image samples;

[0103] The model driving module 64 is configured to drive a target three-dimensional model based on the gesture key point information and the three-dimensional hand mesh.

[0104] Correspondingly, if Figure 7 As shown, Figure 7 : is a block diagram of another model training device for gesture image processing according to an exemplary embodiment of the present specification, the device comprising:

[0105] A sample acquisition module 71 is used to acquire a gesture image and a corresponding three-dimensional hand mesh, wherein the gesture image is marked with gesture key points;

[0106] a first loss module 72 for obtaining image features of the gesture image, inputting the image features into a key point detection network to obtain predicted key points, and obtaining first differences between the predicted key points and the marked gesture key points;

[0107] a second loss module 73, configured to input the image features and the predicted key points into a 3D reconstruction network to obtain a predicted hand mesh, and acquire a second difference between the predicted hand mesh and the 3D hand mesh;

[0108] The joint training module 74 is used to construct a loss function based on the first difference and the second difference, and use the loss function to jointly train the key point detection network and the three-dimensional reconstruction network.

[0109] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0110] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0111] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] Other embodiments of the present invention will readily occur to those skilled in the art upon consideration of the present invention and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present invention being indicated by the following claims.

[0113] It should be understood that the present description is not limited to the exact structure that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.

[0114] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A gesture image processing method, characterized in that: include: Obtaining image features of the target gesture image; the image features include a first feature output by the Mth layer of the feature extraction network and a second feature output by the Nth layer of the feature extraction network, where M is less than N; Inputting the second feature into a key point detection network to perform key point detection to obtain gesture key point information; Inputting the first feature and gesture key point information into a 3D reconstruction network for 3D reconstruction to obtain a 3D hand mesh; the key point detection network and the 3D reconstruction network are jointly trained based on multiple gesture image samples; Driving a target three-dimensional model based on the gesture key point information and the three-dimensional hand mesh; The three-dimensional reconstruction network includes a first sub-network and a second sub-network; inputting the first feature and gesture key point information into the three-dimensional reconstruction network for three-dimensional reconstruction to obtain a three-dimensional hand mesh includes: Inputting the first feature and the gesture key point information into a first sub-network to generate a reconstructed mesh feature, and constructing an initial hand mesh based on the reconstructed mesh feature; The initial hand mesh, the first feature, and the reconstructed mesh feature are input into a second sub-network to generate residual information, and the initial hand mesh is adjusted based on the residual information to obtain a three-dimensional hand mesh.

2. The method according to claim 1, characterized in that The driving of the target three-dimensional model based on the gesture key point information and the three-dimensional hand mesh includes: Inputting the second feature into a gesture segmentation network to obtain a gesture mask; A target three-dimensional model is driven based on the gesture key point information, the three-dimensional hand mesh, and the gesture mask.

3. The method according to claim 1, characterized in that The residual information is obtained by inputting the combined feature into the second sub-network; the combined feature is obtained by converting the initial hand mesh, the first feature and the reconstructed mesh feature into a specified dimension and then superimposing them.

4. The method according to claim 1, wherein The reconstructed grid features input to the second sub-network include the third features output by the S-th layer of the first sub-network and the fourth features output by the T-th layer of the first sub-network, where S is less than T.

5. A gesture image processing method, characterized in that: include: Obtaining a gesture image and a corresponding three-dimensional hand mesh, wherein the gesture image is marked with gesture key points; Obtaining image features of the gesture image, the image features comprising a first feature output by an Mth layer of a feature extraction network and a second feature output by an Nth layer of the feature extraction network, where M is less than N; Inputting the second feature into a key point detection network to obtain a predicted key point, and obtaining a first difference between the predicted key point and the marked gesture key point; Inputting the first feature and the predicted key point into a three-dimensional reconstruction network to obtain a predicted hand mesh, and acquiring a second difference between the predicted hand mesh and the three-dimensional hand mesh; constructing a loss function based on the first difference and the second difference, and jointly training the key point detection network and the 3D reconstruction network using the loss function; The three-dimensional reconstruction network includes a first sub-network and a second sub-network; inputting the first feature and the predicted key point into the three-dimensional reconstruction network to obtain a predicted hand mesh includes: Inputting the first feature and the predicted key point into a first sub-network to generate a reconstructed mesh feature, and constructing an initial hand mesh based on the reconstructed mesh feature; The initial hand mesh, the first feature, and the reconstructed mesh feature are input into a second sub-network to generate residual information, and the initial hand mesh is adjusted based on the residual information to obtain a predicted hand mesh.

6. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 5 when executing the program.

7. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images