Hand image generation method, apparatus, and device

Abstract

The present disclosure provides a hand image generation method, device and equipment, relates to the field of artificial intelligence, specifically to the field of image processing, deep learning and computer vision, and can be applied to metaverse, digital human and the like. The method comprises: obtaining a target hand three-dimensional key point and a reference hand image; generating a target hand posture feature based on the target hand three-dimensional key point; generating a first complete hand texture map based on the reference hand image; and generating a target hand image based on the target hand posture feature and the first complete hand texture map. The embodiment can generate diversified and high-fidelity hand images.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, specifically to the fields of image processing, deep learning, and computer vision, and can be applied to scenarios such as metaverse and digital humans. Background Technology

[0002] With the arrival of the metaverse era, human society has entered a stage of rapid development of virtualization. The development of virtualization is inseparable from Virtual Reality (VR) and Augmented Reality (AR) technologies, and three-dimensional gesture interaction is a crucial interaction method in VR and AR technologies.

[0003] Currently, the core technology of 3D gesture interaction is based on detecting hand joints using deep learning algorithms. Since the essence of deep learning algorithms is to fully train on a dataset, the foundation for supporting accurate and real-time 3D gesture interaction technology is to provide a large number of high-quality hand datasets for different application scenarios. Hand data augmentation methods in related technologies are mainly divided into two categories: one is to augment hand data through image degradation functions and rigid transformations; the other is to synthesize hand data using existing 3D hand models. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, storage medium, and program product for generating hand images.

[0005] In a first aspect, embodiments of this disclosure propose a method for generating a hand image, comprising: acquiring three-dimensional key points of a target hand and a reference hand image; generating target hand pose features based on the three-dimensional key points of the target hand; generating a first complete hand texture map based on the reference hand image; and generating a target hand image based on the target hand pose features and the first complete hand texture map.

[0006] Secondly, embodiments of this disclosure provide a hand image generation apparatus, comprising: an acquisition module configured to acquire three-dimensional key points of a target hand and a reference hand image; a first generation module configured to generate target hand pose features based on the three-dimensional key points of the target hand; a second generation module configured to generate a first complete hand texture map based on the reference hand image; and a target hand image generated based on the target hand pose features and the first complete hand texture map.

[0007] Thirdly, embodiments of this disclosure provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first aspect.

[0008] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform a method as described in any implementation of the first aspect.

[0009] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0010] The hand image generation method provided in this disclosure can generate diverse and highly realistic target hand images by changing the reference hand image to provide different hand textures and by changing the three-dimensional key points of the target hand to provide different hand poses, thereby enhancing the richness of existing hand images and improving the accuracy of gesture estimation.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. Wherein:

[0013] Figure 1 This is a flowchart of an embodiment of the hand image generation method according to the present disclosure;

[0014] Figure 2 This is a flowchart of yet another embodiment of the hand image generation method according to the present disclosure;

[0015] Figure 3 This is a structural diagram of a hand image generation network;

[0016] Figure 4 This is a structural diagram of the adaptive alignment module;

[0017] Figure 5 This is a flowchart of another embodiment of the hand image generation method according to the present disclosure;

[0018] Figure 6 This is a schematic diagram of cyclic generative training;

[0019] Figure 7 This is a schematic diagram of a structure of an embodiment of the hand image generation apparatus according to the present disclosure;

[0020] Figure 8This is a block diagram of an electronic device used to implement the hand image generation method of the embodiments of this disclosure. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] Figure 1 A flow 100 of an embodiment of a hand image generation method according to the present disclosure is shown. The hand image generation method includes the following steps:

[0024] Step 101: Obtain the 3D key points of the target hand and the reference hand image.

[0025] In this embodiment, the entity executing the hand image generation method can acquire the three-dimensional key points of the target hand and a reference hand image.

[0026] The target hand's 3D keypoints can be the 3D coordinates of various key points on the target hand, used to control the hand pose in the generated target hand image. By setting different target hand 3D keypoints, target hand images with different poses can be generated, thus expanding the diversity of hand images. The reference hand image can be a hand image with any pose, providing hand texture to guide the generation of the target hand image. Since the target hand image references the hand texture of this reference hand image, to improve the realism of the generated target hand image, an image of a real hand can be acquired to obtain the reference hand image.

[0027] Step 102: Generate the target hand pose features based on the three-dimensional key points of the target hand.

[0028] In this embodiment, the aforementioned execution entity can generate target hand posture features based on the target hand's three-dimensional key points.

[0029] Since the 3D keypoints of the target hand can control the hand pose in the generated target hand image, feature extraction from the 3D keypoints of the target hand can yield the target hand pose features. These target hand pose features can be the characteristics of the hand pose in the target hand image.

[0030] Step 103: Generate a first complete hand texture map based on the reference hand image.

[0031] In this embodiment, the aforementioned execution entity can generate a first complete hand texture map based on a reference hand image.

[0032] Since a portion of the reference hand is visible in the reference hand image, the texture of the visible portion of the reference hand can be extracted. Texture completion based on this visible texture yields the complete texture of the reference hand, i.e., the first complete hand texture map. This first complete hand texture map can display the 3D texture of the reference hand in image form, clearly showing the details of the back and palm textures, thus better guiding the generation of augmented images of hands in any pose.

[0033] Step 104: Generate a target hand image based on the target hand pose features and the first complete hand texture map.

[0034] In this embodiment, the aforementioned execution entity can generate a target hand image based on the target hand pose features and the first complete hand texture map.

[0035] Typically, the pose of the target hand can be determined based on its pose features, and the texture of the target hand can be determined based on a first complete hand texture map, thus enabling the generation of a target hand image. For example, feature extraction is performed on the first complete hand texture map to obtain its features; the target hand pose features are then fused with the first complete hand texture map features to obtain fused features; and the target hand image is generated based on these fused features. Since the fused features contain both texture and pose, the target hand image generated based on the fused features possesses both the hand pose provided by the target hand's 3D keypoints and the hand texture provided by the reference hand image.

[0036] In some embodiments, annotated target hand images can be obtained by puncturing the target hand image using 3D keypoints. Typically, by varying the reference hand image to provide different hand textures and by changing the target hand's 3D keypoints to provide different hand poses, diverse and highly realistic target hand images can be generated. Furthermore, the generated target hand images can directly use the target hand's 3D keypoints as annotations without needing to be re-annotated, enabling rapid generation of annotated target hand images. This not only improves the efficiency of annotated target hand image generation but also reduces its cost.

[0037] In some embodiments, labeled target hand images are added to the training sample set; the model is trained using the training sample set to obtain a hand 3D keypoint detection model. The training samples in the training sample set are hand images labeled with hand 3D keypoints. Because a large number of high-quality training samples are added to the training sample set, the detection performance of the trained hand 3D keypoint detection model is improved.

[0038] The hand image generation method provided in this disclosure can generate diverse and highly realistic target hand images by changing the reference hand image to provide different hand textures and by changing the three-dimensional key points of the target hand to provide different hand poses, thereby enhancing the richness of existing hand images and improving the accuracy of gesture estimation.

[0039] Continue to refer to Figure 2 This illustrates a flow 200 of yet another embodiment of the hand image generation method according to the present disclosure. The hand image generation method includes the following steps:

[0040] Step 201: Obtain the 3D key points of the target hand and the reference hand image.

[0041] In this embodiment, the specific operation of step 201 has been described. Figure 1 The steps in step 101 of the illustrated embodiment are described in detail and will not be repeated here.

[0042] Step 202: Input the 3D key points of the target hand into the pose extraction network to obtain the pose features of the target hand.

[0043] In this embodiment, the execution entity of the hand image generation method can input the three-dimensional key points of the target hand into a pose extraction network to obtain the target hand pose features. These target hand pose features can be characteristics of the hand pose in the target hand image.

[0044] Step 203: Input the reference hand image into the texture extraction network to obtain the reference hand texture map.

[0045] In this embodiment, the execution entity can input a reference hand image into a texture map extraction network to obtain a reference hand texture map. The reference hand texture map can include the texture of the visible portion of the reference hand.

[0046] Step 204: Input the reference hand texture map into the texture completion network to obtain the first complete hand texture map.

[0047] In this embodiment, the aforementioned execution entity can input a reference hand texture map into a texture completion network to obtain a first complete hand texture map. This first complete hand texture map can include the complete texture of the reference hand, displaying the three-dimensional texture of the reference hand in image form. It can clearly show the details of the texture of the back and palm of the hand, thereby better guiding the generation of augmented images of the hand in any pose.

[0048] Here, a hand image generation network can be used to generate hand images. The hand image generation network can include two branches: a pose extraction branch and a texture map extraction and completion branch. The input to the pose extraction branch is the 3D keypoints of the target hand, mainly used to control the hand pose in the generated target hand image. The input to the texture map extraction and completion branch is a reference hand image. The texture map extraction network extracts the texture of the visible parts of the reference hand, and then the texture map completion network completes the texture of the reference hand to obtain the full texture of the reference hand.

[0049] Step 205: Input the first complete hand texture map into the feature extraction network to obtain the first complete hand texture map features.

[0050] In this embodiment, the execution entity can input the first complete hand texture image into the feature extraction network to obtain the first complete hand texture image features. The feature extraction network can be used to extract image features.

[0051] Step 206: Calculate the deformation matrix based on the target hand pose features and the first complete hand texture map features.

[0052] In this embodiment, the execution entity can calculate the deformation matrix based on the target hand pose features and the first complete hand texture map features.

[0053] In this process, the first complete hand texture map and the target hand's 3D key points are processed through their respective feature extraction networks to obtain the first complete hand texture map features and the target hand pose features. Then, the deformation matrix of the first complete hand texture map features and the target hand pose features is calculated. The deformation matrix encodes the texture required to obtain the target hand pose from the first complete hand texture map. The deformation matrix represents the deformation from the first complete hand texture map to the corresponding image of the target hand pose, and its training adopts an adaptive approach without explicit supervision.

[0054] Step 207: Use the deformation matrix to deform the features of the first complete hand texture map to obtain the deformed features.

[0055] In this embodiment, the aforementioned execution entity can use a deformation matrix to deform the features of the first complete hand texture map to obtain deformed features.

[0056] Since the deformation matrix encodes the texture required to obtain the target hand pose from the first complete hand texture map, the deformation matrix is ​​used to perform deformation operations on the features of the first complete hand texture map, and the resulting deformed features contain the texture of the target hand.

[0057] Step 208: Combine the deformation features with the target hand pose features to obtain the fused features.

[0058] In this embodiment, the aforementioned execution entity can combine the deformation features with the target hand pose features to obtain fused features. The fused features include both texture and pose.

[0059] Step 209: Input the fused features into the decoding network to obtain the target hand image.

[0060] In this embodiment, the aforementioned execution entity can input the fused features into the decoding network to obtain the target hand image.

[0061] Since the fused features contain both texture and pose, the target hand image generated based on the fused features has both the hand pose provided by the 3D key points of the target hand and the hand texture provided by the reference hand image.

[0062] from Figure 2 It can be seen from this that, with Figure 1 Compared to the corresponding embodiments, the hand image generation method in this embodiment emphasizes the image generation step and the feature alignment step in its flow 200. Therefore, the scheme described in this embodiment utilizes a hand image generation network to generate hand images, improving image generation efficiency. The pose extraction branch of the hand image generation network controls the hand pose features in the generated target hand image, while the texture map extraction and completion branch extracts the texture map of the visible portion of the reference hand and completes it to obtain a complete texture map of the reference hand. The texture map is further input into a feature extraction network for feature extraction and then fused with the hand pose features. Furthermore, an adaptive alignment module is introduced during the fusion process to ensure that the palm and back of the hand textures are generated in the correct positions within the hand image. Finally, the fused features are fed into a decoding network to obtain the final target hand image.

[0063] Figure 3 The structure diagram of the hand image generation network is shown. Figure 3As shown, the hand image generation network can include two branches. One is a pose extraction branch: the 3D keypoints of the target hand are input into the pose extraction network, and the target hand pose features are output. The other is a texture extraction and completion branch: a reference hand image is input into the texture extraction network, and a reference hand texture image is output. This reference texture image is then further input into the texture completion network, and a first complete hand texture image is output. The first complete hand texture image is input into the feature extraction network, and the first complete hand texture image features are output. The target hand pose features and the first complete hand texture image features are aligned and fused to obtain fused features. The fused features are input into the decoding network, and the target hand image is output. The target hand image possesses both the hand pose provided by the target hand's 3D keypoints and the hand texture provided by the reference hand image.

[0064] To ensure that the palm and back of the hand textures are generated in the correct positions in the hand image, an adaptive alignment module is introduced during the feature fusion process. Figure 4 A structural diagram of the feature adaptive alignment module is shown. Figure 4 As shown, the first complete hand texture map and the target hand's 3D key points are processed through their respective feature extraction networks to obtain the first complete hand texture map feature f_s and the target hand pose feature f_t. Then, the deformation matrix E is calculated using the first complete hand texture map feature f_s and the target hand pose feature f_t. The deformation matrix E encodes the texture required to obtain the target hand pose from the first complete hand texture map. The deformation matrix E is used to deform the first complete hand texture map feature f_s, resulting in the deformed feature f_w. The deformed feature f_w contains the texture of the target hand. The deformed feature f_w is combined with the target hand pose feature f_t to obtain the fused feature f_a. The fused feature f_a contains both texture and pose.

[0065] Further reference Figure 5 The diagram illustrates a flow 500 of another embodiment of a hand image generation method according to the present disclosure. This hand image generation method includes the following steps:

[0066] Step 501: Obtain the 3D key points of the target hand and the reference hand image.

[0067] Step 502: Input the 3D key points of the target hand into the pose extraction network to obtain the pose features of the target hand.

[0068] Step 503: Input the reference hand image into the texture extraction network to obtain the reference hand texture map.

[0069] Step 504: Input the reference hand texture map into the texture completion network to obtain the first complete hand texture map.

[0070] Step 505: Input the first complete hand texture map into the feature extraction network to obtain the first complete hand texture map features.

[0071] Step 506: Calculate the deformation matrix based on the target hand pose features and the first complete hand texture map features.

[0072] Step 507: Use the deformation matrix to deform the features of the first complete hand texture map to obtain the deformed features.

[0073] Step 508: Combine the deformation features with the target hand pose features to obtain the fused features.

[0074] Step 509: Input the fused features into the decoding network to obtain the target hand image.

[0075] In this embodiment, the specific operations of steps 501-509 have been described. Figure 2 Steps 201-209 in the illustrated embodiments are described in detail and will not be repeated here.

[0076] Step 510: Input the target hand image into the texture extraction network to obtain the target hand texture map.

[0077] In this embodiment, the execution entity can input the target hand image into a texture extraction network to obtain a target hand texture map. The texture extraction network can be used to extract the texture of the visible parts of the target hand.

[0078] Step 511: Input the target hand texture map into the texture completion network to obtain the second complete hand texture map.

[0079] In this embodiment, the execution entity can input the target hand texture map into a texture completion network to obtain a second complete hand texture map. The texture completion network can be used to complete the texture of the target hand based on the visible parts of the target hand.

[0080] Step 512: Calculate the loss based on the first complete hand texture map and the second complete hand texture map.

[0081] In this embodiment, the execution entity can calculate the loss based on the first complete hand texture map and the second complete hand texture map.

[0082] Since the target hand image has the hand texture provided by the reference hand image, the target hand image and the reference hand image correspond to the same set of textures. That is, the first complete hand texture map and the second complete hand texture map are equal. Here, based on the first complete hand texture map and the second complete hand texture map, the loss is calculated, which can supervise the learning of the texture extraction network and the texture completion network.

[0083] Step 513: optimizing the texture extraction network and the texture completion network based on the loss.

[0084] In this embodiment, the above-mentioned execution subject may optimize the texture extraction network and the texture completion network based on the loss. Specifically, parameters of the texture extraction network and the texture completion network are adjusted based on the loss until convergence.

[0085] From Figure 5 it can be seen that compared with the Figure 2 corresponding embodiment, the flow 500 of the hand image generation method in this embodiment adds a cyclic generative training step. Therefore, to ensure that the generated target hand image has similar texture to the reference hand image, the solution described in this embodiment adopts a cyclic generative training method. After the target hand image is obtained through the hand image generation network, the target hand image is further fed into the texture map extraction network and the texture map completion network in the hand image generation network to obtain a second complete hand texture map. Based on the principle that the first complete hand texture map and the second complete hand texture map are equal, the texture map extraction network and the texture map completion network are optimized.

[0086] Figure 6 shows a schematic diagram of cyclic generative training. As Figure 6 shown, after the target hand image is obtained through the hand image generation network, the target hand image is further fed into the texture map extraction network and the texture map completion network to obtain a corresponding texture completion map, which is recorded as texture completion Figure 1 . The texture completion map generated from the reference hand image is recorded as texture completion Figure 2 . Since the two correspond to the same set of texture, texture completion Figure 1 and texture completion Figure 2 should be equal. Based on the above consideration, an L1 loss is calculated through the two to optimize the texture map extraction network and the texture map completion network.

[0087] With further reference to Figure 7 , as an implementation of the method shown in the above figures, the present disclosure provides an embodiment of a hand image generation apparatus, which corresponds to the method embodiment shown in Figure 1 , and the apparatus can be specifically applied to various electronic devices.

[0088] As Figure 7As shown, the hand image generation device 700 of this embodiment may include: an acquisition module 701, a first generation module 702, a second generation module 703, and a third generation module 704. The acquisition module 701 is configured to acquire three-dimensional key points of the target hand and a reference hand image; the first generation module 702 is configured to generate target hand pose features based on the three-dimensional key points of the target hand; the second generation module 703 is configured to generate a first complete hand texture map based on the reference hand image; and the third generation module 704 is configured to generate a target hand image based on the target hand pose features and the first complete hand texture map.

[0089] In this embodiment, the specific processing of the acquisition module 701, the first generation module 702, the second generation module 703, and the third generation module 704 in the hand image generation device 700, and the resulting technical effects, can be found in the following references: Figure 1 The relevant descriptions of steps 101-104 in the corresponding embodiments will not be repeated here.

[0090] In some optional implementations of this embodiment, the third generation module 704 includes: an extraction submodule configured to extract features from the first complete hand texture map to obtain the first complete hand texture map features; a fusion submodule configured to fuse the target hand pose features with the first complete hand texture map features to obtain fused features; and a generation submodule configured to generate a target hand image based on the fused features.

[0091] In some optional implementations of this embodiment, the first generation module 702 is further configured to: input the three-dimensional key points of the target hand into the pose extraction network to obtain the pose features of the target hand.

[0092] In some optional implementations of this embodiment, the second generation module 703 is further configured to: input a reference hand image into a texture extraction network to obtain a reference hand texture map; and input the reference hand texture map into a texture completion network to obtain a first complete hand texture map.

[0093] In some optional implementations of this embodiment, the extraction submodule is further configured to: input the first complete hand texture map into the feature extraction network to obtain the first complete hand texture map features.

[0094] In some optional implementations of this embodiment, the generation submodule is further configured to input the fused features into the decoding network to obtain the target hand image.

[0095] In some optional implementations of this embodiment, the fusion submodule is further configured to: calculate a deformation matrix based on the target hand pose features and the first complete hand texture map features; deform the first complete hand texture map features using the deformation matrix to obtain deformed features; and combine the deformed features with the target hand pose features to obtain fused features.

[0096] In some optional implementations of this embodiment, the hand image generation device 700 further includes: an extraction module configured to input a target hand image into a texture extraction network to obtain a target hand texture map; a completion module configured to input the target hand texture map into a texture completion network to obtain a second complete hand texture map; a calculation module configured to calculate a loss based on the first complete hand texture map and the second complete hand texture map; and an optimization module configured to optimize the texture extraction network and the texture completion network based on the loss.

[0097] In some optional implementations of this embodiment, the hand image generation device 700 further includes an annotation module configured to annotate the target hand image using three-dimensional key points of the target hand to obtain an annotated target hand image.

[0098] In some optional implementations of this embodiment, the hand image generation device 700 further includes: an addition module configured to add the labeled target hand image to the training sample set, wherein the training samples in the training sample set are hand images labeled with three-dimensional key points of the hand; and a training module configured to train the model using the training sample set to obtain a three-dimensional key point detection model of the hand.

[0099] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0100] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0101] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0102] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0103] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0104] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the hand image generation method. For example, in some embodiments, the hand image generation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the hand image generation method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the hand image generation method by any other suitable means (e.g., by means of firmware).

[0105] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0107] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0109] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0110] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0111] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0112] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating a hand image, comprising: Acquire the 3D key points of the target hand and a reference hand image; Based on the three-dimensional key points of the target hand, the target hand posture features are generated; Based on the reference hand image, a first complete hand texture map is generated; Based on the target hand pose features and the first complete hand texture map, a target hand image is generated; The step of generating a first complete hand texture map based on the reference hand image includes: The reference hand image is input into a texture map extraction network to obtain a reference hand texture map; The reference hand texture map is input into the texture completion network to obtain the first complete hand texture map.

2. The method according to claim 1, wherein, The step of generating a target hand image based on the target hand pose features and the first complete hand texture map includes: Feature extraction is performed on the first complete hand texture map to obtain the features of the first complete hand texture map; The target hand pose features are fused with the features of the first complete hand texture map to obtain fused features; Based on the fusion features, the target hand image is generated.

3. The method according to claim 2, wherein, The step of generating the target hand pose based on the target hand's three-dimensional key points includes: The three-dimensional key points of the target hand are input into the pose extraction network to obtain the pose features of the target hand.

4. The method according to claim 3, wherein, The step of extracting features from the first complete hand texture map to obtain the features of the first complete hand texture map includes: The first complete hand texture map is input into the feature extraction network to obtain the features of the first complete hand texture map.

5. The method according to claim 4, wherein, The step of generating a target hand image based on the fused features includes: The fused features are input into the decoding network to obtain the target hand image.

6. The method according to claim 2, wherein, The step of fusing the target hand pose features with the first complete hand texture map features to obtain fused features includes: Based on the target hand pose features and the features of the first complete hand texture map, the deformation matrix is ​​calculated; The deformation matrix is ​​used to deform the features of the first complete hand texture map to obtain deformed features; The deformation feature is combined with the target hand posture feature to obtain the fused feature.

7. The method according to claim 3, wherein, The method further includes: The target hand image is input into the texture extraction network to obtain the target hand texture map; The target hand texture map is input into the texture completion network to obtain a second complete hand texture map; The loss is calculated based on the first complete hand texture map and the second complete hand texture map; Based on the loss, the texture extraction network and the texture completion network are optimized.

8. The method according to any one of claims 1-7, wherein, The method further includes: The target hand image is annotated using the three-dimensional key points of the target hand to obtain an annotated target hand image.

9. The method according to claim 8, wherein, The method further includes: The labeled target hand image is added to the training sample set, wherein the training samples in the training sample set are hand images labeled with three-dimensional key points of the hand; The model was trained using the training sample set to obtain a three-dimensional key point detection model for the hand.

10. A hand image generation device, comprising: The acquisition module is configured to acquire the 3D key points of the target hand and a reference hand image; The first generation module is configured to generate target hand pose features based on the target hand's three-dimensional key points; The second generation module is configured to generate a first complete hand texture map based on the reference hand image; The third generation module is configured to generate a target hand image based on the target hand pose features and the first complete hand texture map; The second generation module is further configured to: The reference hand image is input into a texture map extraction network to obtain a reference hand texture map; The reference hand texture map is input into the texture completion network to obtain the first complete hand texture map.

11. The apparatus according to claim 10, wherein, The third generation module includes: The extraction submodule is configured to extract features from the first complete hand texture map to obtain the features of the first complete hand texture map; The fusion submodule is configured to fuse the target hand pose features with the first complete hand texture map features to obtain fused features; The generation submodule is configured to generate a target hand image based on the fused features.

12. The apparatus according to claim 11, wherein, The first generation module is further configured to: The three-dimensional key points of the target hand are input into the pose extraction network to obtain the pose features of the target hand.

13. The apparatus according to claim 12, wherein, The extraction submodule is further configured to: The first complete hand texture map is input into the feature extraction network to obtain the features of the first complete hand texture map.

14. The apparatus according to claim 13, wherein, The generation submodule is further configured to: The fused features are input into the decoding network to obtain the target hand image.

15. The apparatus according to claim 11, wherein, The fusion submodule is further configured to: Based on the target hand pose features and the features of the first complete hand texture map, the deformation matrix is ​​calculated; The deformation matrix is ​​used to deform the features of the first complete hand texture map to obtain deformed features; The deformation feature is combined with the target hand posture feature to obtain the fused feature.

16. The apparatus according to claim 12, wherein, The device further includes: The extraction module is configured to input the target hand image into the texture extraction network to obtain a target hand texture map; The completion module is configured to input the target hand texture map into the texture completion network to obtain a second complete hand texture map; The calculation module is configured to calculate the loss based on the first complete hand texture map and the second complete hand texture map; An optimization module is configured to optimize the texture extraction network and the texture completion network based on the loss.

17. The apparatus according to any one of claims 10-16, wherein, The device further includes: The annotation module is configured to annotate the target hand image using the three-dimensional key points of the target hand, thereby obtaining an annotated target hand image.

18. The apparatus according to claim 17, wherein, The device further includes: The module is configured to add the labeled target hand image to the training sample set, wherein the training samples in the training sample set are hand images labeled with three-dimensional key points of the hand. The training module is configured to train the model using the training sample set to obtain a three-dimensional key point detection model for the hand.

19. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

20. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-9.

21. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images