Construction method of three-dimensional hand vertex optimization model based on diffusion noise

Abstract

This application proposes a method for constructing a three-dimensional hand vertex optimization model based on diffusion noise, including the following steps: constructing a three-dimensional hand vertex optimization framework, obtaining the global features, first vertex features and time step features of the hand image, and inputting them into the feature fusion self-attention module to denoise the vertex features; weighted fusion of the initial vertex features and the denoised vertex features in the weighted vertex balance module to obtain the optimized vertex features; constructing a loss function based on the optimized vertex features and the corresponding real vertex features, and adjusting the parameters in the three-dimensional hand vertex optimization framework based on the results of the loss function to obtain the three-dimensional hand vertex optimization model. This solution enables the model to capture the spatial relationship and time step information of each vertex by gradually adding noise and denoising to the real vertex features during the training process, effectively solving the modeling deviation caused by local information loss.

Description

Technical Field

[0001] The present application relates to the field of machine learning, and in particular to a method for constructing a three-dimensional hand vertex optimization model based on diffusion and noise. Background Art

[0002] In recent years, 3D hand reconstruction technology has attracted much attention due to its wide application in cutting-edge fields such as action recognition, virtual reality, and human-computer interaction. In the field of 3D hand modeling, the existing mainstream methods are mainly based on the hand parameter optimization model (MANO) and the neural radiance field (NeRF) technology. The hand parametric modeling method based on the MANO model relies on the common deep learning network to predict the 3D joint points and surface structure of the hand from 2D images or video data. It has become an important means to achieve hand posture reconstruction. This method provides high flexibility and model interpretability through the parametric expression of hand shape and posture, making it widely used in gesture recognition and dynamic hand tracking. However, the MANO model still has certain limitations in complex gesture details and local accuracy. Since local features are often lost due to noise interference, posture changes, occlusion, etc., when using the MANO model to recognize complex gestures, the generated vertex positions will be inaccurate due to the lack of local features.

[0003] The same is true for NeRF. When recognizing complex gestures, the self-occlusion phenomenon between fingers will also lead to the loss of local features, making it impossible to accurately identify the vertex position. Therefore, how to efficiently and accurately recognize the vertex position of complex gestures is a problem that the existing technology urgently needs to solve. Summary of the invention

[0004] An embodiment of the present application provides a method for constructing a three-dimensional hand vertex optimization model based on diffusion noising. By gradually adding noise and denoising to real vertex features during the training process, the model can capture the spatial relationship and time step information of each vertex, effectively solving the modeling deviation caused by local information loss.

[0005] In a first aspect, an embodiment of the present application provides a method for constructing a three-dimensional hand vertex optimization model, the method comprising:

[0006] Constructing a three-dimensional hand vertex optimization framework, wherein the three-dimensional hand vertex optimization framework includes a feature fusion self-attention module and a weighted vertex balancing module;

[0007] Acquire at least one hand image and real vertex features corresponding to each hand image, extract global features and initial vertex features of each hand image, preset a time step, and encode the time step to obtain a time step feature;

[0008] Based on a preset time step, the real vertex features of each hand image are gradually denoised to obtain a noisy vertex feature, and the noisy vertex feature of each hand image is concatenated with the initial vertex feature to obtain a first vertex feature, and the global feature, the first vertex feature and the time step feature corresponding to the same hand image are used as training features;

[0009] In the feature fusion self-attention module, the global feature, the first vertex feature and the time step feature in the training feature are added together and then multi-head self-attention calculation is performed to obtain a first result, and then the first result is denoised to obtain a denoised vertex feature, and in the weighted vertex balancing module, the initial vertex feature and the denoised vertex feature are weightedly fused to obtain an optimized vertex feature;

[0010] A loss function is constructed according to the optimized vertex features and the corresponding real vertex features, and the parameters in the three-dimensional hand vertex optimization framework are adjusted according to the result of the loss function. When the value of the loss function meets the set conditions, the parameters are saved to obtain a three-dimensional hand vertex optimization model. The three-dimensional hand vertex optimization model uses the initial vertex features of the hand image as the first vertex features, and combines the corresponding global features and time step features as input.

[0011] In a second aspect, an embodiment of the present application provides an application method of a three-dimensional hand vertex optimization model, including:

[0012] Obtaining initial vertex features and global features of the hand image to be optimized, obtaining time step features, and inputting the initial vertex features, global features and time step features into a three-dimensional hand vertex optimization model to obtain optimized vertex features;

[0013] Constructing a standardized hand model, matching the optimized vertex features with the standardized hand model to obtain joint rotation information and personalized hand shape information, and then generating a transformation matrix from the joint rotation information and the personalized hand shape information;

[0014] A standardized hand skeleton is defined, vertices of the standardized hand model are bound to the standardized hand skeleton to obtain a first three-dimensional hand model, and a transformation matrix is ​​used to drive the vertices of the first three-dimensional hand model to move to obtain a second three-dimensional hand model;

[0015] The vertices of the second three-dimensional hand model are connected into a triangular mesh to obtain the final three-dimensional hand geometric model.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute a method for constructing a three-dimensional hand vertex optimization model.

[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process, and the process includes a method for constructing a three-dimensional hand vertex optimization model.

[0018] The main contributions and innovations of the present invention are as follows:

[0019] In the embodiment of the present application, the operation of gradually adding noise to the real vertex features is adopted in the feature fusion self-attention module, which can enhance the robustness of the model, increase data diversity and help the model learn feature invariance, and linearly transform the splicing result through a linear transformation matrix, so that the self-attention operation is applied to local features and global information, and the discard operation is also used to prevent overfitting. The entire feature fusion self-attention module calculates multi-head self-attention and gradually denoises so that the model can capture the spatial relationship and time step information of each vertex, thereby improving the accuracy and consistency of hand modeling, and effectively solving the modeling deviation caused by local information loss; in the weighted vertex balancing module, a multi-layer perceptron is used to gradually process the denoised vertex features and the initial vertex features, so that dynamically adjusted adaptive denoising weights and initial weights can be generated according to the differences in position and detail capture of different vertices, and the vertex position is adjusted according to the weight to achieve accurate optimization of the vertex position, remove residual noise, and ensure the consistency and stability of the vertex in complex scenes. The weighted vertex balancing module significantly enhances the robustness of the hand model to noise and deformation, and improves the accuracy and detail reproduction ability of three-dimensional hand reconstruction.

[0020] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 is a schematic diagram of a training process of a three-dimensional hand vertex optimization model according to an embodiment of the present application;

[0023] Figure 2 is a structural schematic diagram of a feature fusion self-attention module according to an embodiment of the present application;

[0024] Figure 3 is a structural schematic diagram of a weighted vertex balancing module according to an embodiment of the present application;

[0025] Figure 4 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0027] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0028] Embodiment 1

[0029] The embodiment of the present application provides a method for constructing a three-dimensional hand vertex optimization model based on diffusion noise. By gradually adding noise and denoising to the real vertex features during the training process, the model can capture the spatial relationship and time step information of each vertex, and effectively solves the modeling deviation caused by local information loss. Specifically, reference Figure 1 , the method comprising:

[0030] Constructing a three-dimensional hand vertex optimization framework, wherein the three-dimensional hand vertex optimization framework includes a feature fusion self-attention module and a weighted vertex balancing module;

[0031] Acquire at least one hand image and real vertex features corresponding to each hand image, extract global features and initial vertex features of each hand image, preset a time step, and encode the time step to obtain a time step feature;

[0032] Based on a preset time step, the real vertex features of each hand image are gradually denoised to obtain a noisy vertex feature, and the noisy vertex feature of each hand image is concatenated with the initial vertex feature to obtain a first vertex feature, and the global feature, the first vertex feature and the time step feature corresponding to the same hand image are used as training features;

[0033] In the feature fusion self-attention module, the global feature, the first vertex feature and the time step feature in the training feature are added together and then multi-head self-attention calculation is performed to obtain a first result, and then the first result is denoised to obtain a denoised vertex feature, and in the weighted vertex balancing module, the initial vertex feature and the denoised vertex feature are weightedly fused to obtain an optimized vertex feature;

[0034] A loss function is constructed according to the optimized vertex features and the corresponding real vertex features, and the parameters in the three-dimensional hand vertex optimization framework are adjusted according to the result of the loss function. When the value of the loss function meets the set conditions, the parameters are saved to obtain a three-dimensional hand vertex optimization model. The three-dimensional hand vertex optimization model uses the initial vertex features of the hand image as the first vertex features, and combines the corresponding global features and time step features as input.

[0035] In some specific embodiments, a hand image is input into a pre-trained feature extraction model to obtain hand image features and hand parameters, the hand image features are first reduced in dimension by linear transformation to obtain a reduced dimension result, and then a nonlinear activation function is used to output the reduced dimension result to obtain a global feature, and the hand parameters are input into a pre-trained MANO model to obtain initial vertex features.

[0036] Furthermore, the feature extraction model used in this solution is a pre-trained FastViT model. The hand parameters include the shape parameters and joint parameters of the hand. The hand parameters are output to the MANO model to obtain the initial vertex features corresponding to the hand image.

[0037] Specifically, since the three-dimensional model corresponding to the hand image is composed of multiple vertices, the initial vertex features described in this solution are the position coordinates between each vertex and the connection relationship between the vertices.

[0038] In some embodiments, the formula for reducing the dimension of the hand image features by linear transformation is expressed as follows:

[0039]

[0040] in, is the dimensionality reduction result, is the linear transformation matrix, is the hand image feature.

[0041] Furthermore, the formula for obtaining the global features by using a nonlinear activation function to output the dimension reduction results is as follows:

[0042]

[0043] in, is a global feature, is a nonlinear activation function, This is the result of dimensionality reduction.

[0044] Specifically, since hand image features usually have very high dimensions and contain a lot of information, such as pixel values, color channels, textures and other details, in the subsequent calculation process, high-dimensional data will cause a sharp increase in computational costs. Therefore, this scheme uses linear transformation to reduce the dimension, which can convert these complex high-dimensional data into low-dimensional representations, thereby greatly reducing the amount of calculation and improving the training speed and inference speed of the model.

[0045] In some embodiments, a diffusion model is used to gradually add Gaussian noise to the real vertex features at a preset time step.

[0046] Specifically, by scanning the hand or using a motion capture system to accurately obtain the real vertex features corresponding to the hand image, more diverse training samples can be generated by gradually adding noise to the real vertex features, enriching the distribution of training data. In addition, by gradually adding noise to the real vertex features, the model can learn to ignore irrelevant changes caused by noise and focus on the essential part of the real vertices, which helps the model extract more representative and stable features.

[0047] In some embodiments, the time step is encoded by sine-cosine encoding to obtain a time step feature, which is expressed as follows:

[0048]

[0049] in, is the time step feature, d is the embedding dimension, i represents each position of the embedding vector, and in sine-cosine encoding, sine is used for even positions and cosine is used for odd positions.

[0050] Specifically, sine and cosine encoding can give each time step a unique position feature, allowing the model to better understand the evolution of hand movements in time series.

[0051] In some specific embodiments, the noisy vertex features are concatenated with the initial vertex features by linear projection, and the formula for concatenating the noisy vertex features of each hand image with the initial vertex features to obtain the first vertex features is expressed as follows:

[0052]

[0053] in, represents the first vertex feature, is the initial vertex feature, To add noisy vertex features, Represents a linear projection.

[0054] Specifically, concatenating the initial vertex features with the noisy vertex features allows the model to better balance the reliance on the original information and the processing of noise interference, thereby more accurately optimizing the hand vertices.

[0055] In this solution, the structure of the feature fusion self-attention module is as follows Figure 2 As shown, the feature fusion self-attention module includes a multi-head self-attention mechanism layer and a vertex attention layer. The global feature, the first vertex feature and the time step feature are added to obtain a fused feature. The fused feature is first input into the multi-head self-attention layer for multi-head self-attention calculation, and the calculation results of all attention heads are concatenated to obtain a first result. The first result is then input into the vertex attention layer for denoising to obtain a denoised vertex feature.

[0056] Specifically, the formula for adding the global feature, the first vertex feature, and the time step feature is expressed as:

[0057]

[0058] in, To fusion features, is the first vertex feature, is a global feature, is the time step feature.

[0059] In some specific embodiments, the fusion features are respectively represented by the weight matrix W q , W k , W v Get the query Q, key K and value V, and then use Q, K, and V to calculate the calculation result of each attention head. The formula is as follows:

[0060]

[0061]

[0062] in, is the calculation result of the attention head, Q i , K i 、V i Represent the query, key, and value matrices for each header, respectively.

[0063] Furthermore, the calculation results of each attention head are first concatenated to obtain a concatenated result, and then the concatenated result is linearly transformed by a linear transformation matrix to obtain a first result, which is expressed as follows:

[0064]

[0065] in, For the first result, Represents the calculation results of different attention heads, is the linear transformation matrix.

[0066] Specifically, this scheme linearly transforms the splicing result through a linear transformation matrix, so as to apply the self-attention operation to local features and global information, where the local features are the vertex features of the hand, and the global features are the global features and time step features.

[0067] Furthermore, the vertex attention layer denoises the first result in a layer-by-layer optimization manner to obtain a denoised result, and then discards the denoised result to obtain a discarded result. Finally, the discarded result is added to the fusion feature and normalized and output to obtain the denoised vertex feature. The formula is as follows:

[0068]

[0069] in, To denoise vertex features, It means layer-by-layer optimization. Indicates a discard operation. For the first result, To fusion features, Indicates normalization.

[0070] Specifically, the discard operation is used to prevent overfitting. The entire feature fusion self-attention module calculates multi-head self-attention and step-by-step denoising, enabling the model to capture the spatial relationship and time step information of each vertex, thereby improving the accuracy and consistency of hand modeling and effectively solving the modeling deviation caused by local information loss.

[0071] In this solution, the structure of the weighted vertex balancing module is as follows: Figure 3 As shown, in the weighted vertex balancing module, the denoised vertex features and the initial vertex features are dimensionally concatenated to obtain a second result, the second result is passed through a multi-layer perceptron to obtain initial weights and denoising weights, and finally the initial vertex features and the denoised vertex features are weightedly fused according to the initial weights and denoising weights to obtain the optimized vertex features.

[0072] Specifically, in the weighted vertex balancing module, the denoised vertex features With initial vertex features By splicing in dimension, the vertex coordinates change from 3 to 6, and the second result is obtained .

[0073] Specifically, the formula for processing the second result by the multi-layer perceptron is expressed as follows:

[0074]

[0075]

[0076] in, To denoise vertex features The denoising weight, is the initial vertex feature The initial weights of MLP are multi-layer perceptrons. For the second result, is a non-linear activation function.

[0077] Furthermore, the multilayer perceptron consists of a first fully connected layer, a GELU activation function, a Dropout layer, a second fully connected layer, and a Sigmoid activation function connected in series.

[0078] Specifically, the formula for obtaining the optimized vertex features by weighted fusion of the initial vertex features and the denoising vertex features according to the initial weights and the denoising weights is as follows:

[0079]

[0080] in, To optimize vertex features, To denoise vertex features The denoising weight, is the initial vertex feature The initial weight of .

[0081] Specifically, the weighted vertex balancing module uses a multi-layer perceptron to gradually process the denoising vertex features and the initial vertex features, so that it can generate dynamically adjusted adaptive denoising weights and initial weights according to the differences in position and detail capture of different vertices, and adjust the vertex positions according to the weights to achieve precise optimization of the vertex positions, remove residual noise, and ensure the consistency and stability of the vertices in complex scenes. The weighted vertex balancing module significantly enhances the robustness of the hand model to noise and deformation, and improves the accuracy and detail reproduction ability of three-dimensional hand reconstruction.

[0082] In some specific embodiments, the first vertex feature is obtained by concatenating the noisy vertex feature of each hand image with the initial vertex feature only during the training process of the model, and the initial vertex feature is used as the first vertex feature in the application stage after the training of the three-dimensional hand vertex optimization model is completed.

[0083] Embodiment 2

[0084] Based on the same concept, the present application also proposes an application method of a three-dimensional hand vertex optimization model, including:

[0085] Obtaining initial vertex features and global features of the hand image to be optimized, obtaining time step features, and inputting the initial vertex features, global features and time step features into a three-dimensional hand vertex optimization model to obtain optimized vertex features;

[0086] Constructing a standardized hand model, matching the optimized vertex features with the standardized hand model to obtain joint rotation information and personalized hand shape information, and then generating a transformation matrix from the joint rotation information and the personalized hand shape information;

[0087] A standardized hand skeleton is defined, vertices of the standardized hand model are bound to the standardized hand skeleton to obtain a first three-dimensional hand model, and a transformation matrix is ​​used to drive the vertices of the first three-dimensional hand model to move to obtain a second three-dimensional hand model;

[0088] The vertices of the second three-dimensional hand model are connected into a triangular mesh to obtain the final three-dimensional hand geometric model.

[0089] In some specific embodiments, matching is performed by minimizing the difference between the optimized vertex features and the standard vertices on the corresponding standard hand model, thereby obtaining joint rotation information and personalized hand shape information.

[0090] In some specific embodiments, a standardized hand skeleton is predefined using a MANO model, and the Linear Blend Skinning (LBS) technology of the MANO model is used to bind the vertices of the standardized hand model to the standardized hand skeleton, and the movement of the vertices of the standardized hand model is driven by the transformation matrix T to generate vertices v of the hand mesh. By loading the predefined triangle patch connection relationship in the MANO model, the generated vertices v are connected into a triangular mesh, and finally a complete three-dimensional geometric model of the hand is constructed.

[0091] Embodiment 3

[0092] This embodiment also provides an electronic device, referring to Figure 4 , comprises a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above method embodiments.

[0093] Specifically, the processor 402 may include a central processing unit (CPU), a graphics processing unit (GPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0094] Among them, the memory 404 may include a large capacity memory 404 for data or instructions. For example, but not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 404 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 404 may be inside or outside the data processing device. In a specific embodiment, the memory 404 is a non-volatile memory. In a specific embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. In appropriate circumstances, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM may be a fast page mode dynamic random access memory 404 (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0095] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .

[0096] The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement any one of the methods for constructing a three-dimensional hand vertex optimization model based on diffusion and noise in the above embodiments.

[0097] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .

[0098] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0099] The input and output device 408 is used to input or output information. In this embodiment, the input information may be initial vertex features, time step features, global features, etc., and the output information may be optimized vertex features, etc.

[0100] Optionally, in this embodiment, the processor 402 may be configured to perform the following steps through a computer program:

[0101] Constructing a three-dimensional hand vertex optimization framework, wherein the three-dimensional hand vertex optimization framework includes a feature fusion self-attention module and a weighted vertex balancing module;

[0102] Acquire at least one hand image and real vertex features corresponding to each hand image, extract global features and initial vertex features of each hand image, preset a time step, and encode the time step to obtain a time step feature;

[0103] Based on a preset time step, the real vertex features of each hand image are gradually denoised to obtain a noisy vertex feature, and the noisy vertex feature of each hand image is concatenated with the initial vertex feature to obtain a first vertex feature, and the global feature, the first vertex feature and the time step feature corresponding to the same hand image are used as training features;

[0104] In the feature fusion self-attention module, the global feature, the first vertex feature and the time step feature in the training feature are added together and then multi-head self-attention calculation is performed to obtain a first result, and then the first result is denoised to obtain a denoised vertex feature, and in the weighted vertex balancing module, the initial vertex feature and the denoised vertex feature are weightedly fused to obtain an optimized vertex feature;

[0105] A loss function is constructed according to the optimized vertex features and the corresponding real vertex features, and the parameters in the three-dimensional hand vertex optimization framework are adjusted according to the result of the loss function. When the value of the loss function meets the set conditions, the parameters are saved to obtain a three-dimensional hand vertex optimization model. The three-dimensional hand vertex optimization model uses the initial vertex features of the hand image as the first vertex features, and combines the corresponding global features and time step features as input.

[0106] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0107] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0108] Embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, at this point, it should be noted that, for example, Figure 4Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.

[0109] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for constructing a three-dimensional hand vertex optimization model, characterized in that: The following steps are involved: Constructing a three-dimensional hand vertex optimization framework, wherein the three-dimensional hand vertex optimization framework includes a feature fusion self-attention module and a weighted vertex balancing module; Acquire at least one hand image and real vertex features corresponding to each hand image, extract global features and initial vertex features of each hand image, preset a time step, and encode the time step to obtain a time step feature; Based on a preset time step, the real vertex features of each hand image are gradually denoised to obtain a noisy vertex feature, and the noisy vertex feature of each hand image is concatenated with the initial vertex feature to obtain a first vertex feature, and the global feature, the first vertex feature and the time step feature corresponding to the same hand image are used as training features; In the feature fusion self-attention module, the global feature, the first vertex feature and the time step feature in the training feature are added together and then multi-head self-attention calculation is performed to obtain a first result, and then the first result is denoised to obtain a denoised vertex feature, and in the weighted vertex balancing module, the initial vertex feature and the denoised vertex feature are weightedly fused to obtain an optimized vertex feature; A loss function is constructed according to the optimized vertex features and the corresponding real vertex features, and the parameters in the three-dimensional hand vertex optimization framework are adjusted according to the result of the loss function. When the value of the loss function meets the set conditions, the parameters are saved to obtain a three-dimensional hand vertex optimization model. The three-dimensional hand vertex optimization model uses the initial vertex features of the hand image as the first vertex features, and combines the corresponding global features and time step features as input.

2. The method for constructing a three-dimensional hand vertex optimization model according to claim 1, characterized in that: The hand image is input into the pre-trained feature extraction model to obtain the hand image features and hand parameters. The hand image features are first reduced in dimension by linear transformation, and then output using a nonlinear activation function to obtain the global features. The hand parameters are input into the pre-trained MANO model to obtain the initial vertex features.

3. The method for constructing a three-dimensional hand vertex optimization model according to claim 1, characterized in that: The feature fusion self-attention module includes a multi-head self-attention mechanism layer and a vertex attention layer. The global feature, the first vertex feature and the time step feature are added together to obtain a fused feature. The fused feature is first input into the multi-head self-attention layer for multi-head self-attention calculation, and the calculation results of all attention heads are concatenated to obtain a first result. The first result is then input into the vertex attention layer for denoising to obtain a denoised vertex feature.

4. The method for constructing a three-dimensional hand vertex optimization model according to claim 3, characterized in that: First, the calculation results of each attention head are spliced ​​to obtain a spliced ​​result, and then the spliced ​​result is linearly transformed by a linear transformation matrix to obtain a first result.

5. The method for constructing a three-dimensional hand vertex optimization model according to claim 3, characterized in that: The vertex attention layer denoises the first result in a layer-by-layer optimization manner to obtain a denoised result, then discards the denoised result to obtain a discarded result, and finally adds the discarded result to the fusion feature and normalizes the output to obtain the denoised vertex feature.

6. The method for constructing a three-dimensional hand vertex optimization model according to claim 1, characterized in that: In the weighted vertex balancing module, the denoised vertex features and the initial vertex features are dimensionally spliced ​​to obtain the second result. The second result is passed through a multi-layer perceptron to obtain the initial weights and denoising weights. Finally, the initial vertex features and the denoised vertex features are weightedly fused according to the initial weights and denoising weights to obtain the optimized vertex features.

7. An application method of a three-dimensional hand vertex optimization model, characterized in that: include: Obtaining initial vertex features and global features of the hand image to be optimized, and obtaining time step features, and inputting the initial vertex features, global features, and time step features into a three-dimensional hand vertex optimization model constructed by any method of claims 1-6 to obtain optimized vertex features; Constructing a standardized hand model, matching the optimized vertex features with the standardized hand model to obtain joint rotation information and personalized hand shape information, and then generating a transformation matrix from the joint rotation information and the personalized hand shape information; A standardized hand skeleton is defined, vertices of the standardized hand model are bound to the standardized hand skeleton to obtain a first three-dimensional hand model, and a transformation matrix is ​​used to drive the vertices of the first three-dimensional hand model to move to obtain a second three-dimensional hand model; The vertices of the second three-dimensional hand model are connected into a triangular mesh to obtain the final three-dimensional hand geometric model.

8. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the method for constructing a three-dimensional hand vertex optimization model according to any one of claims 1-6.

9. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, and the process includes a method for constructing a three-dimensional hand vertex optimization model according to any one of claims 1-6.

Images