Hand image restoration method, device and equipment

Abstract

The invention discloses a hand image restoration method, device and equipment, which are used for restoring a hand image and improving the image quality of image generation. The method comprises the following steps: intercepting a hand image to be restored from a source image; obtaining a structure graph pair of the hand image to be restored, wherein the structure graph pair comprises a two-dimensional key point graph and a three-dimensional reconstruction graph; retrieving a target structure chart pair with the highest similarity with the structure chart pair of the hand image to be restored in a hand image retrieval library; extracting a hand two-dimensional structure feature and a hand three-dimensional structure feature of the target structure map pair, extracting a hand skin color feature of a hand region of the to-be-restored hand image, and extracting an image background feature of the to-be-restored hand image; and inputting the hand two-dimensional structure feature, the hand three-dimensional structure feature, the hand skin color feature and the image background feature into an image generation model to obtain a repaired hand image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a method, apparatus, and device for hand image restoration. Background Technology

[0002] With the rapid development of diffusion models, image generation has become increasingly realistic and usable. However, current image generation schemes still have some problems when generating images containing human hands. For example, the generated images may contain hand deformities, such as extra or missing fingers, or unreasonable finger lengths; the hands may be blurry, failing to generate clear and aesthetically pleasing hands; and there may be abnormal interactions between the hand and objects. These issues with the quality of generated images containing hands severely restrict the large-scale adoption of such applications. Summary of the Invention

[0003] In view of this, embodiments of this application provide a method, apparatus, and device for repairing hand images to improve the image quality of the generated images.

[0004] To address the above problems, the technical solutions provided in this application are as follows:

[0005] In a first aspect, embodiments of this application provide a method for restoring a hand image, the method comprising:

[0006] Extract the image of the hand to be repaired from the source image;

[0007] Obtain a pair of structural images of the hand image to be repaired, the pair of structural images including a two-dimensional key point image and a three-dimensional reconstruction image;

[0008] Search the hand image retrieval database for the target structural image pair with the highest similarity to the structural image pair of the hand to be repaired;

[0009] Extract the two-dimensional and three-dimensional structural features of the hand from the target structure image pair, extract the skin color features of the hand region in the hand image to be repaired, and extract the image background features of the hand image to be repaired;

[0010] The two-dimensional structural features of the hand, the three-dimensional structural features of the hand, the skin color features of the hand, and the background features of the image are input into the image generation model to obtain the repaired hand image.

[0011] Secondly, embodiments of this application provide a hand image restoration device, the device comprising:

[0012] The cropping unit is used to crop out the hand image to be repaired from the source image;

[0013] The first acquisition unit is used to acquire a pair of structural images of the hand image to be repaired, the pair of structural images including a two-dimensional key point image and a three-dimensional reconstruction image;

[0014] The retrieval unit is used to retrieve, in the hand image retrieval database, the target structural image pair with the highest similarity to the structural image pair of the hand to be repaired;

[0015] The first extraction unit is used to extract the two-dimensional and three-dimensional structural features of the hand from the target structure image pair, extract the skin color features of the hand region of the hand image to be repaired, and extract the image background features of the hand image to be repaired.

[0016] The repair unit is used to input the two-dimensional structural features of the hand, the three-dimensional structural features of the hand, the skin color features of the hand, and the background features of the image into the image generation model to obtain the repaired hand image.

[0017] Thirdly, embodiments of this application provide a hand image restoration device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the hand image restoration method as described above.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform the hand image restoration method described above.

[0019] Therefore, the embodiments of this application have the following beneficial effects:

[0020] In this embodiment, the hand image to be restored is extracted from the source image, and a two-dimensional keypoint map and a three-dimensional reconstructed map of the hand image to be restored are obtained. The most similar target structure map pair from the same real hand image is retrieved from a hand image retrieval library established from real hand images, serving as the control condition for the image generation model. The two-dimensional and three-dimensional structural features of the hand corresponding to the target structure map pair, the skin color features of the hand region in the hand image to be restored, and the background features of the corresponding image of the hand image to be restored are input into the image generation model to obtain the restored hand image. Since the target structure map pair corresponds to a real hand image, inputting its features into the image generation model can effectively improve the ability of the image generation model to generate hands. Simultaneously, the image generation model also references the skin color features and background features of the hand, thereby producing a higher quality restored hand image. Attached Figure Description

[0021] Figure 1 A schematic diagram illustrating an exemplary application scenario provided in this application embodiment;

[0022] Figure 2 A flowchart illustrating a hand image restoration method provided in this application embodiment;

[0023] Figure 3 This is a schematic diagram of the hand image restoration method in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the training process of the image generation model in the embodiments of this application;

[0025] Figure 5 This is a schematic diagram of the training process of the image generation model in the embodiments of this application;

[0026] Figure 6 A schematic diagram of a hand image restoration device provided in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of a device provided in an embodiment of this application. Detailed Implementation

[0028] To make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the embodiments of this application will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0029] To facilitate understanding and explanation of the technical solutions provided in the embodiments of this application, the background technology of the embodiments of this application will be described first below.

[0030] With the rapid development of diffusion models, image generation has become increasingly realistic and usable. However, current image generation schemes suffer from the technical problem of low image quality when generating images containing human hands.

[0031] Based on this, embodiments of this application provide a method, apparatus, and device for hand image restoration, which generates hand images based on retrieval enhancement to improve the image quality of the generated images, particularly the image quality of the generated images including human hand images.

[0032] To facilitate understanding of the hand image restoration method provided in the embodiments of this application, the following is combined with... Figure 1 The example scenario is shown below. See also... Figure 1 As shown in the figure, this figure is a schematic diagram of an exemplary application scenario provided in the embodiments of this application.

[0033] The embodiments of this application can be applied to image generation clients, other image processing clients, or image processing servers. The embodiments of this application do not limit the executing entity of the provided hand image restoration method.

[0034] In practical applications, a large number of real human hand images can be collected in advance, and key point detectors and 3D hand reconstruction methods can be used to process the real hand images to obtain corresponding 2D (two-dimensional) key point maps and 3D (three-dimensional) reconstruction maps, so as to establish a hand image retrieval library.

[0035] The process begins by acquiring a source image, which can be an image containing hand anomalies. The image near the hand region is then cropped to obtain the hand image to be repaired. A keypoint detector and a 3D hand reconstruction method are used to obtain 2D keypoint maps and 3D reconstructed images of the hand image to be repaired. Using these images, a search is conducted in a hand image retrieval database to find the most similar real hand image. The 2D keypoint map and 3D reconstructed image of this real hand image are then used as the target 2D keypoint map and target 3D reconstructed image, forming a target structure map pair.

[0036] The process involves extracting 2D keypoint features of the hand from the target image and 3D structural features from the reconstructed 3D image. Segmenting the hand region of the image to be restored and extracting its features yields skin tone characteristics, preserving the skin tone and style of the hand. Furthermore, the process extracts background features from the occlusion image of the hand to be restored, preserving the original background, features of the person in the original image, and the style of the original image.

[0037] The image generation model is input with the two-dimensional structural features of the hand, the three-dimensional structural features of the hand, the skin color features of the hand, and the background features of the image to guide the image generation process and obtain a restored hand image with normal structure and aesthetic posture.

[0038] Those skilled in the art will understand that Figure 1 The schematic diagram shown is merely one example in which embodiments of this application can be implemented. The scope of application of the embodiments of this application is not limited by any aspect of this framework.

[0039] To facilitate understanding of the embodiments of this application, the following description, in conjunction with the accompanying drawings, illustrates a hand image restoration method provided by the embodiments of this application.

[0040] See Figure 2 As shown, this figure is a flowchart of a hand image restoration method provided in an embodiment of this application. Figure 2 As shown, the method may include S201-S205:

[0041] S201: Extract the image of the hand to be repaired from the source image.

[0042] First, a source image is acquired. The source image can be a generated image with hand abnormalities, or any other image that includes the human hand region. This application embodiment does not limit the method or specific type of source image acquisition. If image restoration is required, the image near the hand region in the source image is cropped to obtain the hand image to be restored.

[0043] In one possible implementation, before executing S201, the following can also be done:

[0044] Input the source image into the image confidence model to obtain the confidence parameter; if the confidence parameter is less than the threshold, execute S201 to extract the hand image to be repaired from the source image and follow up with subsequent steps.

[0045] In other words, the source image can be input into the image confidence model, which detects the location of the hand in the source image and provides a confidence parameter for each hand. The confidence parameter can be used to characterize the image quality of the hand in the source image. If the confidence parameter is below a threshold, it indicates that the image quality is low and needs repair; in this case, the hand image repair method provided in this embodiment continues to be executed. Otherwise, no repair is needed. The image confidence model can be a convolutional neural network; this embodiment does not limit the model structure of the image confidence model.

[0046] Pre-judging image quality using an image confidence model and then repairing it when the image quality is low can save computational resources.

[0047] S202: Obtain the structural image pair of the hand image to be repaired, which includes a two-dimensional key point image and a three-dimensional reconstruction image.

[0048] The 2D keypoint map and 3D reconstructed map of the hand image to be restored are extracted, and the 2D keypoint map and 3D reconstructed map are grouped to form a structure map pair. For example, a keypoint detector can be used to obtain the 2D keypoint map, and a 3D reconstructed map can be obtained using a 3D hand reconstruction method. This application does not limit the specific implementation method for obtaining the 2D keypoint map and 3D reconstructed map.

[0049] S203: Retrieve the target structural image pair with the highest similarity to the structural image pair of the hand to be repaired in the hand image retrieval database.

[0050] In practical applications, a large number of real human hand images can be collected in advance, and 2D keypoint maps and 3D reconstructed maps can be extracted from each real hand image. For example, keypoint detectors and 3D hand reconstruction methods can be used for extraction. The 2D keypoint map and 3D reconstructed map of each real hand image serve as the structural map pair for that real hand image. This allows the establishment of a hand image retrieval library, which includes a large number of structural map pairs (2D keypoint maps and 3D reconstructed maps) of real hand images.

[0051] The 2D key point map and 3D reconstructed map of the hand image to be repaired are searched in the hand image retrieval database to obtain the most similar target structure map pair belonging to the same real hand image. The target structure map pair includes the target 2D key point map and the target 3D reconstructed map.

[0052] This application embodiment provides a specific implementation method for S203 to retrieve the target structural image pair with the highest similarity to the structural image pair of the hand to be repaired in the hand image retrieval library. Please refer to the description of the following embodiments, which will not be repeated here.

[0053] In this embodiment, the target structure map retrieved from the hand image retrieval database is used as the input to the image generation model, i.e., as the control condition for the image generation model. The reason and advantage of this approach are as follows:

[0054] Reconstructing the correct hand structure from abnormal hand images is extremely difficult, and even if the correct hand structure can be reconstructed, there is a significant probability of abnormal hand pose. Using the reconstructed image from the abnormal hand image as the control condition for the image generation model easily leads to low-quality hand restoration images. Furthermore, the hand structure and pose reconstructed from abnormal hand images differ somewhat from the distribution of the structure and pose of real human hands. Since most image generation models are trained using real human hand images, this difference in distribution reduces the quality of the generated images. Therefore, in this embodiment, retrieving similar hand reconstruction results from a hand image retrieval library built from real human hand images as the control condition for the image generation model can effectively improve the model's ability to generate hands and produce higher-quality hand restoration images.

[0055] S204: Extract the two-dimensional and three-dimensional structural features of the hand from the target structure image pair, extract the skin color features of the hand region in the hand image to be repaired, and extract the background features of the hand image to be repaired.

[0056] After obtaining the target structure map pair, further image feature enhancement is required because the image generation model needs to use image features (such as feature vectors). Specifically, the two-dimensional structural features of the hand in the target 2D keypoint map of the target structure map pair are extracted to represent the 2D structure of the hand; and the three-dimensional structural features of the hand in the target 3D reconstructed map of the target structure map pair are extracted to represent the 3D structure of the hand.

[0057] The image of the hand to be restored primarily focuses on the hand region, but may also include other parts such as the background. The goal is to segment the hand region, extract its skin tone features, and thus preserve the hand's skin tone and style. The image can have different styles, such as realistic, anime, or those using specific filters. The skin tone features characterize both the skin tone and style of the hand region.

[0058] Furthermore, by occluding the hand region in the image of the hand to be restored, an occlusion map of the hand image can be obtained. The background features of the occlusion map are then extracted to preserve the original image background, the original person's ID / IP address, and the original image style. The image background may also include relevant parts of the human body, such as clothing and posture, which can constitute the original person's ID / IP address. Similarly, the image background can also have different styles. Therefore, the image background features can characterize the original image background, the person, and the style.

[0059] It is understood that the extraction of two-dimensional hand structural features, three-dimensional hand structural features, hand skin color features, and image background features can be performed separately, and there is no need to limit the order in which different features are extracted. Furthermore, the specific implementation method of the feature extraction step S204 provided in this application embodiment can be found in the description of subsequent embodiments, and will not be repeated here.

[0060] S205: Input the two-dimensional structural features of the hand, the three-dimensional structural features of the hand, the skin color features of the hand, and the background features of the image into the image generation model to obtain the repaired hand image.

[0061] By inputting the two-dimensional structural features of the hand, the three-dimensional structural features of the hand, the skin color features of the hand, and the background features of the image into the image generation model, a restored hand image with normal structure and aesthetic posture can be obtained.

[0062] In practical applications, the image generation model can be a diffusion model. The diffusion model works by continuously adding Gaussian noise to corrupt the training data, and then reversing this noise process to learn and recover the data. When using a diffusion model, randomly sampled noise can be fed into it, and the model generates data by learning the denoising process. In this embodiment, noise, along with two-dimensional hand structural features, three-dimensional hand structural features, hand skin color features, and image background features, can be input into the image generation model to generate a restored hand image.

[0063] For details on the specific structure and training method of the image generation model, please refer to the description in the following embodiments.

[0064] In this embodiment, the hand image to be restored is extracted from the source image, and a two-dimensional keypoint map and a three-dimensional reconstructed map of the hand image to be restored are obtained. The most similar target structure map pair from the same real hand image is retrieved from a hand image retrieval library established from real hand images, serving as the control condition for the image generation model. The two-dimensional and three-dimensional structural features of the hand corresponding to the target structure map pair, the skin color features of the hand region in the hand image to be restored, and the background features of the corresponding image of the hand image to be restored are input into the image generation model to obtain the restored hand image. Since the target structure map pair corresponds to a real hand image, inputting its features into the image generation model can effectively improve the ability of the image generation model to generate hands. Simultaneously, the image generation model also references the skin color features and background features of the hand, thereby producing a higher quality restored hand image.

[0065] The following further explains the specific implementation of S203. In one possible implementation, the specific implementation of S203 in retrieving the target structural image pair with the highest similarity to the structural image pair of the hand image to be repaired from the hand image retrieval database may include:

[0066] A1: Determine the first similarity between the 2D keypoint map of the hand image to be repaired and the 2D keypoint map of the real hand image in the hand image retrieval database.

[0067] The 2D keypoint map of the hand image to be repaired is compared with each 2D keypoint map in the hand image retrieval database to obtain the first similarity of each 2D keypoint map in the hand image retrieval database.

[0068] In the first possible implementation of A1, the method includes: normalizing the two-dimensional key point map of the hand image to be repaired and the two-dimensional key point map of the real hand image in the hand image retrieval library, averaging the cosine similarity between each corresponding pixel point, and obtaining the first similarity between the two-dimensional key point map of the hand image to be repaired and the two-dimensional key point map of the real hand image in the hand image retrieval library.

[0069] In the first implementation, retrieval and matching can be performed based on 2D pose. Specifically, the two 2D keypoint images to be compared (the 2D keypoint image of the hand to be repaired and the 2D keypoint image of any real hand image in the hand image retrieval database) are normalized, and the normalized images are of the same size. The cosine similarity of the vectors formed by every two corresponding pixels is calculated, and then the average value is taken to obtain the first similarity of the 2D keypoint images to be compared.

[0070] In a second possible implementation of A1, the method includes: inputting a two-dimensional keypoint map of the hand image to be repaired into a pose detection model to obtain a first pose feature output by the pose detection model; inputting a two-dimensional keypoint map of a real hand image from a hand image retrieval database into the pose detection model to obtain a second pose feature output by the pose detection model; and calculating the similarity between the first pose feature and the second pose feature as the first similarity between the two-dimensional keypoint map of the hand image to be repaired and the two-dimensional keypoint map of the real hand image from the hand image retrieval database.

[0071] In the second implementation, retrieval and matching can be based on a pose detection model. Specifically, the two 2D keypoint images to be compared are input into the pose detection model, and the last layer of features from the model is extracted to obtain the first pose feature and the second pose feature. The similarity between the first pose feature and the second pose feature is calculated as the first similarity of the 2D keypoint images to be compared. The pose detection model can extract features such as pose and gestures from the 2D keypoint images; for example, pose features can represent a "like" gesture, an "OK" gesture, etc.

[0072] In a third possible implementation of A1, the following steps are taken: inputting the two-dimensional keypoint map of the hand image to be restored into the CLIP (Contrastive Language-Image Pre-training) model to obtain the first hand semantic feature output by the CLIP model; inputting the two-dimensional keypoint map of a real hand image from the hand image retrieval database into the CLIP model to obtain the second hand semantic feature output by the CLIP model; and calculating the similarity between the first hand semantic feature and the second hand semantic feature as the first similarity between the two-dimensional keypoint map of the hand image to be restored and the two-dimensional keypoint map of a real hand image from the hand image retrieval database.

[0073] In the third implementation, retrieval matching can be performed based on the CLIP (Contrastive Language-Image Pre-training) model. Specifically, two 2D keypoint images to be compared are input into the CLIP model to obtain first and second hand semantic features. The similarity between the first and second hand semantic features is calculated as the first similarity of the 2D keypoint images to be compared. The CLIP model is a multimodal pre-trained neural network model that learns the matching relationship between image and text vector representations by comparing them. During the training phase, the CLIP model receives a batch of image-text pairs as input, attempting to bring matching image and text vectors closer in a shared semantic space, while pushing mismatched vectors further apart. This learning method allows CLIP to capture the deep semantic connections between images and text. Thus, after inputting the 2D keypoint images into the CLIP model, hand semantic features can be obtained, such as those representing what the hand is doing.

[0074] In practical applications, any one of the three possible implementation methods can be used, or multiple methods can be used. If multiple methods are used, the first similarity obtained from each implementation method needs to be weighted and summed to obtain the final first similarity between the two-dimensional key point map of the hand image to be repaired and the two-dimensional key point map of the real hand image in the hand image retrieval database.

[0075] A2: Determine the second similarity between the 3D reconstructed image of the hand to be repaired and the 3D reconstructed images of real hand images in the hand image retrieval database.

[0076] The 3D reconstructed image of the hand image to be repaired is compared with each 3D reconstructed image in the hand image retrieval database to obtain the first similarity of each 3D reconstructed image in the hand image retrieval database.

[0077] In the first possible implementation of A2, the method includes: normalizing the 3D reconstructed image of the hand image to be repaired and the 3D reconstructed image of the real hand image in the hand image retrieval library, averaging the cosine similarity between corresponding pixels, and obtaining a second similarity between the 3D reconstructed image of the hand image to be repaired and the 3D reconstructed image of the real hand image in the hand image retrieval library.

[0078] In the first implementation, retrieval and matching can be performed based on 3D pose. Specifically, the two 3D reconstructed images to be compared (the 3D reconstructed image of the hand to be repaired and the 3D reconstructed image of any real hand image in the hand image retrieval database) are normalized, and the normalized images are of the same size. The cosine similarity of the vectors formed by every two corresponding pixels is calculated, and then the average value is taken to obtain the second similarity of the 3D reconstructed images to be compared.

[0079] In a second possible implementation of A2, the method includes: inputting the 3D reconstructed image of the hand image to be repaired into the pose detection model to obtain the third pose feature output by the pose detection model; inputting the 3D reconstructed image of a real hand image from the hand image retrieval database into the pose detection model to obtain the fourth pose feature output by the pose detection model; and calculating the similarity between the third pose feature and the fourth pose feature as the second similarity between the 3D reconstructed image of the hand image to be repaired and the 3D reconstructed image of a real hand image from the hand image retrieval database.

[0080] In the second implementation, retrieval and matching can be based on a pose detection model. Specifically, the two 3D reconstructed images to be compared are input into the pose detection model, and the last layer of features from the pose detection model is extracted to obtain the third and fourth pose features. The similarity between the third and fourth pose features is calculated as the second similarity of the 3D reconstructed images to be compared. The pose detection model can extract features such as pose and gestures from the 3D keypoint map.

[0081] In a third possible implementation of A2, the following steps are taken: inputting the 3D reconstructed image of the hand to be repaired into the CLIP model to obtain the third hand semantic feature output by the CLIP model; inputting the 3D reconstructed image of a real hand image from the hand image retrieval database into the CLIP model to obtain the fourth hand semantic feature output by the CLIP model; and calculating the similarity between the third hand semantic feature and the fourth hand semantic feature as the second similarity between the 3D reconstructed image of the hand to be repaired and the 3D reconstructed image of a real hand image from the hand image retrieval database.

[0082] In the third implementation, retrieval and matching can be performed based on the CLIP model. Specifically, the two 3D reconstructed images to be compared are input into the CLIP model to obtain the semantic features of the third and fourth hands. The similarity between the semantic features of the third and fourth hands is calculated as the second similarity of the 3D reconstructed images to be compared.

[0083] In practical applications, any one of the three possible implementation methods can be used, or multiple methods can be used. If multiple methods are used, the second similarity obtained by each implementation method needs to be weighted and summed to obtain the second similarity between the two-dimensional key point map of the hand image to be repaired and the two-dimensional key point map of the real hand image in the hand image retrieval library.

[0084] A3: The similarity of the real hand image is obtained by weighted summing of the first similarity of the two-dimensional key point map corresponding to the real hand image and the second similarity of the three-dimensional reconstruction map.

[0085] Since the structure diagrams of the final search results need to correspond to the same real hand image, the first similarity and the second similarity corresponding to the same real hand image are weighted and summed to obtain the similarity of each real hand image.

[0086] A4: The two-dimensional key point map and the three-dimensional reconstructed map of the real hand image with the highest similarity are identified as the target structure map pair.

[0087] Finally, the 2D key point map and 3D reconstruction map of the real hand image with the highest similarity were selected as the target structure map pair (target 2D key point map and target 3D reconstruction map).

[0088] In this embodiment, target structure image pairs of real hand images most similar to the hand image to be restored can be retrieved from a hand image retrieval database using different retrieval methods. Image generation is then performed based on the target structure image pairs, making the generated hand restoration image closer to the real hand image and improving image quality.

[0089] In one possible implementation, the specific implementation of extracting the two-dimensional and three-dimensional structural features of the hand from the target structure map pair in S204 may include:

[0090] The target structure map is aligned with the target's two-dimensional key point map and input into the two-dimensional structure extraction model to obtain the two-dimensional structural features of the hand.

[0091] The target structure map is aligned with the target 3D reconstruction map and input into the 3D structure extraction model to obtain the 3D structural features of the hand.

[0092] In practical applications, the 2D structure extraction model (PoseGuider2d) can be a convolutional neural network that extracts features from a 2D keypoint map. It downsamples the input target 2D keypoint map to obtain the 2D structural features of the hand. For example, downsampling the target 2D keypoint map from 512x512 to 64x64 yields the 2D structural features of the hand.

[0093] The 3D structure extraction model (PoseGuider3d) can be a convolutional neural network that extracts features from a 3D reconstructed image. It downsamples the input target 3D reconstructed image to obtain the 3D structural features of the hand. For example, downsampling the target 3D keymap from 512x512 to 64x64 yields the 3D structural features of the hand.

[0094] In this embodiment, the model structure of the two-dimensional structure extraction model and the three-dimensional structure extraction model are not limited, and can be selected according to the actual situation.

[0095] In one possible implementation, the specific implementation of extracting the skin color features of the hand region in the hand image to be repaired in S204 may include:

[0096] The hand region is segmented from the hand image to be repaired, and the hand region is input into the CLIP model to obtain the skin color features of the hand region in the hand image to be repaired.

[0097] This involves segmenting the hand region of the image to be repaired, inputting the hand region into the CLIP model, and obtaining the skin color features of the hand. These skin color features are high-level semantic information that can more accurately describe the information in the image.

[0098] In one possible implementation, the specific implementation of extracting the image background features of the hand image to be repaired in S204 may include:

[0099] Occlude the hand region in the hand image to be repaired to obtain an occlusion map of the hand image to be repaired, or replace the hand region in the hand image to be repaired with the target 3D reconstruction map to obtain an occlusion map of the hand image to be repaired; input the occlusion map of the hand image to be repaired into the reference encoder to obtain the image background features of the hand image to be repaired.

[0100] Occlusion of the hand region in the image of the hand to be restored yields an occlusion map. This occlusion map is then input into a reference encoder to obtain the background features of the image, thus preserving relevant features from the background.

[0101] In another implementation, the hand region in the image of the hand to be restored can be replaced with the target 3D reconstructed image to obtain an occlusion map of the hand image to be restored. This occlusion map is then input into a reference encoder to obtain the background features of the hand image to be restored. This approach allows the image generation model to learn relevant features such as the hand contour earlier, while simultaneously obtaining relevant features from the image background.

[0102] This application does not limit the model structure of ReferenceEncoder; it can be selected according to the actual situation.

[0103] See Figure 3 The diagram illustrates a hand image restoration method according to an embodiment of this application. After obtaining the target structure map pair, the target 2D keypoint map is input into a 2D structure extraction model, and the target 3D keypoint map is input into a 3D structure extraction model. The hand region of the hand image to be restored is input into the CLIP model, and the occlusion map of the hand image to be restored is input into a reference encoder. Two-dimensional hand structure features, three-dimensional hand structure features, hand skin color features, and image background features are extracted respectively.

[0104] In one possible implementation, the image generation model includes a Reference Unet and a Denoising Unet. The Reference Unet and Denoising Unet have the same number of layers, for example, both including n sampling layers and n upsampling layers. The features output from each layer of the Reference Unet are input to the corresponding layer of the Denoising Unet. For example, the features output from the first layer of the Reference Unet are injected into the first layer of the Denoising Unet, the features output from the second layer of the Reference Unet are injected into the second layer of the Denoising Unet, and so on.

[0105] Based on this, in one possible implementation, S205 inputs the two-dimensional structural features of the hand, the three-dimensional structural features of the hand, the skin color features of the hand, and the image background features into the image generation model to obtain the repaired hand image. Specific implementations may include:

[0106] Background features of the image are input into the reference Unet, skin color features of the hand are input into each layer of both the reference Unet and the denoising Unet, and 2D and 3D structural features of the hand are input into the denoising Unet. The restored hand image is generated by decoding the features output by the denoising Unet.

[0107] See also Figure 4 As shown, image background features are only input into the Reference Unet, while hand skin color features are input into each layer of the Reference Unet and the Denoising Unet. The hand 2D structure features, hand 3D structure features, and noise are superimposed and then input into the Denoising Unet. The output of the Denoising Unet is then decoded by the decoder to generate the final repaired hand image.

[0108] This application does not limit the model structure of Reference Unet, Denoising Unet, and Decoder; the appropriate structure can be selected based on the actual situation.

[0109] In this embodiment, Reference Unet and Denoising Unet are used to repair abnormal hand images while preserving information such as background, human features, and skin color of the hand image to be repaired, so as to generate correct hand images.

[0110] This application also provides the training process for the image generation model, see [link to relevant documentation]. Figure 4 The diagram illustrates the training process of the image generation model in an embodiment of this application. The training process of the image generation model may include:

[0111] B1: Obtain the structural image pair of the real hand image to be trained.

[0112] In this embodiment, a large number of real hand images can be acquired as real hand images to be trained. Structural image pairs (2D keypoint maps and 3D reconstructed maps) are extracted from the hand images to be trained. For example, a keypoint detector can be used to acquire 2D keypoint maps, and a 3D reconstructed map can be acquired using a 3D hand reconstruction method.

[0113] B2: Extract the two-dimensional and three-dimensional structural features of the hand to be trained from the structural image of the real hand image, extract the skin color features of the hand region of the real hand image, and extract the background features of the real hand image.

[0114] Similarly, the 2D keypoint map of the real hand image to be trained extracts the 2D structural features of the hand, and the 3D reconstructed image extracts the 3D structural features of the hand. Segmenting the hand region of the hand image to be trained and extracting its features yields the skin color features, thus preserving the skin color and style of the hand. Background features of the hand image to be restored are extracted, preserving the original background, the features of the person in the original image, and the style of the original image.

[0115] In one possible implementation, B2 extracts the structure map of the real hand image to be trained, including the two-dimensional and three-dimensional structural features of the hand to be trained, extracts the skin color features of the hand region in the real hand image to be trained, and extracts the background features of the real hand image to be trained. Specific implementations of these extraction methods may include:

[0116] C1: Input the structural map of the real hand image to be trained into the two-dimensional key point map into the two-dimensional structure extraction model to obtain the two-dimensional structural features of the hand to be trained.

[0117] C2: Input the 3D reconstruction map of the structure map of the real hand image to be trained into the 3D structure extraction model to obtain the 3D structural features of the hand to be trained.

[0118] C3: Segment the target hand region from the real hand image to be trained, input the target hand region into the CLIP model, and obtain the skin color features of the hand region of the real hand image to be trained.

[0119] C4: Occlude any region in the real hand image to be trained to obtain the occlusion map of the real hand image to be trained; input the occlusion map of the real hand image to be trained into the reference encoder to obtain the image background features of the hand image to be repaired.

[0120] See Figure 5 The diagram shown illustrates the training process of another image generation model in an embodiment of this application. Figure 3 Correspondingly, during the training process, the model structures of the 2D structure extraction model, 3D structure extraction model, CLIP model, reference encoder, and image generation model are consistent with those in the above embodiments. For related explanations, please refer to the above embodiments.

[0121] After obtaining the 2D keypoint map of the real hand image to be trained, it is input into the 2D structure extraction model. After obtaining the 3D reconstructed map of the real hand image to be trained, it is input into the 3D structure extraction model. The hand region of the real hand image to be trained is input into the CLIP model, and the occlusion map of the real hand image to be trained is input into the reference encoder. The 2D structure features, 3D structure features, skin color features, and background features of the hand to be trained are extracted respectively.

[0122] B3: Input the two-dimensional structural features of the hand to be trained, the three-dimensional structural features of the hand to be trained, the skin color features of the hand to be trained, and the background features of the image to be trained into the image generation model to obtain the target hand image.

[0123] Similarly, image generation models include Reference Unet and Denoising Unet. Reference Unet and Denoising Unet have the same number of layers, and the features output from each layer of Reference Unet are input to the corresponding layer of Denoising Unet.

[0124] In one possible implementation, B3 inputs the two-dimensional structural features of the hand to be trained, the three-dimensional structural features of the hand to be trained, the skin color features of the hand to be trained, and the background features of the image to be trained into the image generation model to obtain the target hand image. Specific implementations may include:

[0125] The background features of the image to be trained are input into the reference Unet, the skin color features of the hand to be trained are input into each layer of the reference Unet and the denoising Unet, and the two-dimensional structure features and three-dimensional structure features of the hand to be trained are input into the denoising Unet. The target hand image is generated by decoding the features output by the denoising Unet.

[0126] See also Figure 5 As shown, the background features of the image to be trained are only input into the Reference Unet, while the skin color features of the hand to be trained are input into each layer of the Reference Unet and the Denoising Unet. The two-dimensional structural features of the hand to be trained, the three-dimensional structural features of the hand to be trained, and the noise are superimposed and then input into the Denoising Unet. The output of the Denoising Unet is then processed by the Decoder to generate the final target hand image.

[0127] B4: Train the image generation model based on the loss value between the target hand image and the real hand image to be trained.

[0128] Specifically, the model parameters of the reference Une, the denoising Une, the 2D structure extraction model, and the 3D structure extraction model can be adjusted based on the loss value between the target hand image and the real hand image to be trained.

[0129] That is, there is a difference between the generated target hand image and the real hand image to be trained. Based on this difference, the model parameters of the image generation model, the two-dimensional structure extraction model and the three-dimensional structure extraction model are adjusted. Steps B1-B4 are repeatedly executed iteratively to reduce the difference until the preset conditions are met, and the final trained image generation model is obtained.

[0130] In practical applications, the initial model parameters of the 2D and 3D structure extraction models are randomly initialized, while the Reference Unet and Denoising Unet are initialized using the corresponding Unet parameters. The parameters of the ReferenceEncoder, Decoder, and CLIP models (the shaded parts in the figure) are fixed and do not require training.

[0131] In this embodiment, an image generation model can be trained using a real hand image to be trained. The training process is relatively efficient, and the trained image generation model can generate a restored hand image that is closer to the real hand image.

[0132] Based on the hand image restoration method provided in the above-described method embodiments, this application also provides a hand image restoration device, which will be described below with reference to the accompanying drawings.

[0133] See Figure 6 As shown in the figure, this is a schematic diagram of the structure of a hand image restoration device provided in an embodiment of this application. Figure 6 As shown, the hand image restoration device includes:

[0134] The cropping unit 601 is used to crop out the hand image to be repaired from the source image;

[0135] The first acquisition unit 602 is used to acquire a pair of structural images of the hand image to be repaired, the pair of structural images including a two-dimensional key point image and a three-dimensional reconstruction image;

[0136] The retrieval unit 603 is used to retrieve, in the hand image retrieval database, the target structural image pair with the highest similarity to the structural image pair of the hand image to be repaired;

[0137] The first extraction unit 604 is used to extract the two-dimensional and three-dimensional structural features of the hand from the target structure image pair, extract the skin color features of the hand region of the hand image to be repaired, and extract the background features of the image of the hand to be repaired.

[0138] The repair unit 605 is used to input the two-dimensional structural features of the hand, the three-dimensional structural features of the hand, the skin color features of the hand, and the background features of the image into the image generation model to obtain the repaired hand image.

[0139] In one possible implementation, the retrieval unit includes:

[0140] The first determining subunit is used to determine the first similarity between the two-dimensional key point map of the hand image to be repaired and the two-dimensional key point map of the real hand image in the hand image retrieval database;

[0141] The second determining subunit is used to determine the second similarity between the three-dimensional reconstruction image of the hand image to be repaired and the three-dimensional reconstruction image of the real hand image in the hand image retrieval database;

[0142] The calculation subunit is used to weight and sum the first similarity corresponding to the two-dimensional key point map and the second similarity corresponding to the three-dimensional reconstruction map of the same real hand image to obtain the similarity of the real hand image.

[0143] The third determining subunit is used to determine the two-dimensional key point map and the three-dimensional reconstructed map of the real hand image with the highest similarity as the target structure map pair.

[0144] In one possible implementation, the first determining subunit is specifically used for:

[0145] The two-dimensional key point map of the hand image to be repaired is normalized with the two-dimensional key point map of the real hand image in the hand image retrieval library. The cosine similarity between each corresponding pixel is averaged to obtain the first similarity between the two-dimensional key point map of the hand image to be repaired and the two-dimensional key point map of the real hand image in the hand image retrieval library.

[0146] The second determining subunit is specifically used for:

[0147] The 3D reconstructed image of the hand image to be repaired is normalized with the 3D reconstructed image of the real hand image in the hand image retrieval database. The cosine similarity between each corresponding pixel is averaged to obtain the second similarity between the 3D reconstructed image of the hand image to be repaired and the 3D reconstructed image of the real hand image in the hand image retrieval database.

[0148] In one possible implementation, the first determining subunit is specifically used for:

[0149] The two-dimensional key point map of the hand image to be repaired is input into the pose detection model to obtain the first pose feature output by the pose detection model.

[0150] The two-dimensional key point map of real hand images from the hand image retrieval library is input into the pose detection model to obtain the second pose feature output by the pose detection model.

[0151] The similarity between the first pose feature and the second pose feature is calculated as the first similarity between the two-dimensional key point map of the hand image to be repaired and the two-dimensional key point map of the real hand image in the hand image retrieval database.

[0152] The second determining subunit is specifically used for:

[0153] The three-dimensional reconstructed image of the hand to be repaired is input into the pose detection model to obtain the third pose feature output by the pose detection model.

[0154] The three-dimensional reconstructed image of a real hand image from the hand image retrieval database is input into the pose detection model to obtain the fourth pose feature output by the pose detection model.

[0155] The similarity between the third pose feature and the fourth pose feature is calculated as the second similarity between the 3D reconstructed image of the hand image to be repaired and the 3D reconstructed image of a real hand image in the hand image retrieval database.

[0156] In one possible implementation, the first determining subunit is specifically used for:

[0157] The two-dimensional key point map of the hand image to be repaired is input into the contrastive language-image pre-trained CLIP model to obtain the first hand semantic features output by the CLIP model;

[0158] The two-dimensional key point map of real hand images from the hand image retrieval library is input into the CLIP model to obtain the second hand semantic features output by the CLIP model.

[0159] The similarity between the first hand semantic features and the second hand semantic features is calculated as the first similarity between the two-dimensional key point map of the hand image to be repaired and the two-dimensional key point map of the real hand image in the hand image retrieval database.

[0160] The second determining subunit is specifically used for:

[0161] The three-dimensional reconstructed image of the hand to be repaired is input into the CLIP model to obtain the third semantic feature of the hand output by the CLIP model;

[0162] The three-dimensional reconstruction image of a real hand image from the hand image retrieval database is input into the CLIP model to obtain the fourth hand semantic feature output by the CLIP model.

[0163] The similarity between the third hand semantic feature and the fourth hand semantic feature is calculated as the second similarity between the 3D reconstruction map of the hand image to be repaired and the 3D reconstruction map of the real hand image in the hand image retrieval database.

[0164] In one possible implementation, the first extraction unit is specifically used for:

[0165] The target structure map is aligned with the target two-dimensional key point map and input into the two-dimensional structure extraction model to obtain the two-dimensional structure features of the hand.

[0166] The target structure map is aligned with the target 3D reconstruction map and input into the 3D structure extraction model to obtain the 3D structure features of the hand.

[0167] The hand region is segmented from the hand image to be repaired, and the hand region is input into the CLIP model to obtain the skin color features of the hand region of the hand image to be repaired.

[0168] The hand region in the hand image to be repaired is occluded to obtain an occlusion map of the hand image to be repaired; or, the hand region in the hand image to be repaired is replaced with the target 3D reconstruction map to obtain an occlusion map of the hand image to be repaired. The occlusion map of the hand image to be repaired is input into a reference encoder to obtain the image background features of the hand image to be repaired.

[0169] In one possible implementation, the image generation model includes a reference Unet and a denoising Unet, wherein the reference Unet and the denoising Unet have the same number of layers, and the features output by each layer of the reference Unet are input to the corresponding layer of the denoising Unet.

[0170] The repair unit is specifically used for:

[0171] The image background features are input into the reference Unet, the hand skin color features are input into each layer of the reference Unet and the denoising Unet, and the hand two-dimensional structure features and the hand three-dimensional structure features are input into the denoising Unet.

[0172] The repaired hand image is generated by obtaining and decoding the features output by the denoised Unet.

[0173] In one possible implementation, the device further includes:

[0174] The second acquisition unit is used to acquire the structure map pairs of the real hand image to be trained;

[0175] The second extraction unit is used to extract the two-dimensional structural features and three-dimensional structural features of the hand to be trained from the structure map pair of the real hand image to be trained, extract the skin color features of the hand region of the real hand image to be trained, and extract the background features of the real hand image to be trained.

[0176] The obtaining unit is used to input the two-dimensional structural features of the hand to be trained, the three-dimensional structural features of the hand to be trained, the skin color features of the hand to be trained, and the background features of the image to be trained into the image generation model to obtain the target hand image;

[0177] The training unit is used to train the image generation model based on the loss value between the target hand image and the real hand image to be trained.

[0178] In one possible implementation, the second extraction unit is specifically used for:

[0179] The structural map of the real hand image to be trained is aligned with the two-dimensional key point map and input into the two-dimensional structure extraction model to obtain the two-dimensional structural features of the hand to be trained.

[0180] The structural map of the real hand image to be trained is aligned with the 3D reconstruction map and input into the 3D structure extraction model to obtain the 3D structural features of the hand to be trained.

[0181] The target hand region is segmented from the real hand image to be trained, and the target hand region is input into the CLIP model to obtain the skin color features of the hand region of the real hand image to be trained.

[0182] Occlude any region in the real hand image to be trained to obtain an occlusion map of the real hand image to be trained; input the occlusion map of the real hand image to be trained into a reference encoder to obtain the background features of the hand image to be repaired.

[0183] In one possible implementation, the image generation model includes a reference Unet and a denoising Unet, wherein the reference Unet and the denoising Unet have the same number of layers, and the features output by each layer of the reference Unet are input to the corresponding layer of the denoising Unet.

[0184] The obtaining unit is specifically used for:

[0185] The background features of the image to be trained are input into the reference Unet, the skin color features of the hand to be trained are input into each layer of the reference Unet and the denoising Unet, and the two-dimensional structure features of the hand to be trained and the three-dimensional structure features of the hand to be trained are input into the denoising Unet.

[0186] The target hand image is generated by obtaining and decoding the features output by the denoised Unet.

[0187] The training unit is specifically used for:

[0188] Based on the loss value between the target hand image and the real hand image to be trained, the model parameters of the reference Une, the denoising Unet, the two-dimensional structure extraction model, and the three-dimensional structure extraction model are adjusted.

[0189] In one possible implementation, the device further includes:

[0190] The input unit is used to input the source image into the image confidence model to obtain confidence parameters;

[0191] A triggering unit is used to trigger the cropping unit to crop the hand image to be repaired from the source image if the confidence parameter is less than a threshold.

[0192] This application also provides a computer program product, including computer program instructions, which, when executed on a computer, cause the computer to perform the hand image restoration method as described in any of the preceding embodiments.

[0193] Based on the partial image restoration method provided in the above method embodiments, this application also provides an electronic device, including: one or more processors; a storage device storing one or more programs thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the partial image restoration method described in any of the above embodiments.

[0194] The following is for reference. Figure 7 This document illustrates a structural schematic diagram of an electronic device 1300 suitable for implementing embodiments of this application. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Android devices), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs (televisions), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0195] like Figure 7As shown, electronic device 1300 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 1301, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1302 or a program loaded from storage device 1306 into random access memory (RAM) 1303. RAM 1303 also stores various programs and data required for the operation of electronic device 1300. Processing device 1301, ROM 1302, and RAM 1303 are interconnected via bus 1304. Input / output (I / O) interface 1305 is also connected to bus 1304.

[0196] Typically, the following devices can be connected to I / O interface 1305: input devices 1306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1306 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1309. Communication device 1309 allows electronic device 1300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 1300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0197] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1309, or installed from storage device 1306, or installed from ROM 1302. When the computer program is executed by processing device 1301, it performs the functions defined in the methods of embodiments of this application.

[0198] The electronic device provided in this application embodiment belongs to the same inventive concept as the hand image restoration method provided in the above embodiment. Technical details not described in detail in this embodiment can be found in the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0199] Based on the hand image restoration method provided in the above embodiments, this application provides a computer-readable medium storing a computer program thereon, wherein the program, when executed by a processor, implements the hand image restoration method as described in any of the above embodiments.

[0200] It should be noted that the computer-readable medium described in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the embodiments of this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0201] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0202] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0203] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the aforementioned hand image restoration method.

[0204] Computer program code for performing the operations of the embodiments of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0205] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0206] The units described in the embodiments of this application can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0207] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0208] In the context of embodiments of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0209] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0210] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0211] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0212] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0213] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for restoring hand images, characterized in that, The method includes: Extract the image of the hand to be repaired from the source image; Obtain a pair of structural images of the hand image to be repaired, the pair of structural images including a two-dimensional key point image and a three-dimensional reconstruction image; Search the hand image retrieval database for the target structural image pair with the highest similarity to the structural image pair of the hand to be repaired; Extract the two-dimensional and three-dimensional structural features of the hand from the target structure image pair, extract the skin color features of the hand region in the hand image to be repaired, and extract the image background features of the hand image to be repaired; The two-dimensional structural features of the hand, the three-dimensional structural features of the hand, the skin color features of the hand, and the background features of the image are input into the image generation model to obtain the repaired hand image.

2. The method according to claim 1, characterized in that, The step of retrieving the target structural image pair with the highest similarity to the structural image pair of the hand image to be repaired from the hand image retrieval database includes: Determine the first similarity between the two-dimensional key point map of the hand image to be repaired and the two-dimensional key point map of the real hand image in the hand image retrieval database; Determine the second similarity between the 3D reconstructed image of the hand to be repaired and the 3D reconstructed image of a real hand image in the hand image retrieval database; The similarity of the real hand image is obtained by weighted summing of the first similarity corresponding to the two-dimensional key point map and the second similarity corresponding to the three-dimensional reconstruction map of the same real hand image. The two-dimensional key point map and the three-dimensional reconstructed map of the real hand image with the highest similarity are identified as the target structure map pair.

3. The method according to claim 2, characterized in that, Determining the first similarity between the two-dimensional keypoint map of the hand image to be repaired and the two-dimensional keypoint map of a real hand image in the hand image retrieval database includes: The two-dimensional key point map of the hand image to be repaired is normalized with the two-dimensional key point map of the real hand image in the hand image retrieval library. The cosine similarity between each corresponding pixel is averaged to obtain the first similarity between the two-dimensional key point map of the hand image to be repaired and the two-dimensional key point map of the real hand image in the hand image retrieval library. The determination of the second similarity between the 3D reconstructed image of the hand image to be repaired and the 3D reconstructed image of a real hand image in the hand image retrieval database includes: The 3D reconstructed image of the hand image to be repaired is normalized with the 3D reconstructed image of the real hand image in the hand image retrieval database. The cosine similarity between each corresponding pixel is averaged to obtain the second similarity between the 3D reconstructed image of the hand image to be repaired and the 3D reconstructed image of the real hand image in the hand image retrieval database.

4. The method according to claim 2, characterized in that, Determining the first similarity between the two-dimensional keypoint map of the hand image to be repaired and the two-dimensional keypoint map of a real hand image in the hand image retrieval database includes: The two-dimensional key point map of the hand image to be repaired is input into the pose detection model to obtain the first pose feature output by the pose detection model. The two-dimensional key point map of real hand images from the hand image retrieval library is input into the pose detection model to obtain the second pose feature output by the pose detection model. The similarity between the first pose feature and the second pose feature is calculated as the first similarity between the two-dimensional key point map of the hand image to be repaired and the two-dimensional key point map of the real hand image in the hand image retrieval database. The determination of the second similarity between the 3D reconstructed image of the hand image to be repaired and the 3D reconstructed image of a real hand image in the hand image retrieval database includes: The three-dimensional reconstructed image of the hand to be repaired is input into the pose detection model to obtain the third pose feature output by the pose detection model. The three-dimensional reconstructed image of a real hand image from the hand image retrieval database is input into the pose detection model to obtain the fourth pose feature output by the pose detection model. The similarity between the third pose feature and the fourth pose feature is calculated as the second similarity between the 3D reconstructed image of the hand image to be repaired and the 3D reconstructed image of a real hand image in the hand image retrieval database.

5. The method according to claim 2, characterized in that, Determining the first similarity between the two-dimensional keypoint map of the hand image to be repaired and the two-dimensional keypoint map of a real hand image in the hand image retrieval database includes: The two-dimensional key point map of the hand image to be repaired is input into the contrastive language-image pre-trained CLIP model to obtain the first hand semantic features output by the CLIP model; The two-dimensional key point map of real hand images from the hand image retrieval library is input into the CLIP model to obtain the second hand semantic features output by the CLIP model. The similarity between the first hand semantic features and the second hand semantic features is calculated as the first similarity between the two-dimensional key point map of the hand image to be repaired and the two-dimensional key point map of the real hand image in the hand image retrieval database. The determination of the second similarity between the 3D reconstructed image of the hand image to be repaired and the 3D reconstructed image of a real hand image in the hand image retrieval database includes: The three-dimensional reconstruction image of the hand to be repaired is input into the CLIP model to obtain the third semantic features of the hand output by the CLIP model; The three-dimensional reconstruction image of a real hand image from the hand image retrieval database is input into the CLIP model to obtain the fourth hand semantic feature output by the CLIP model. The similarity between the third hand semantic feature and the fourth hand semantic feature is calculated as the second similarity between the 3D reconstruction map of the hand image to be repaired and the 3D reconstruction map of the real hand image in the hand image retrieval database.

6. The method according to claim 1, characterized in that, The steps of extracting the two-dimensional and three-dimensional structural features of the hand from the target structure image pair, extracting the skin color features of the hand region in the hand image to be repaired, and extracting the image background features of the hand image to be repaired include: The target structure map is aligned with the target two-dimensional key point map and input into the two-dimensional structure extraction model to obtain the two-dimensional structure features of the hand. The target structure map is aligned with the target 3D reconstruction map and input into the 3D structure extraction model to obtain the 3D structure features of the hand. The hand region is segmented from the hand image to be repaired, and the hand region is input into the CLIP model to obtain the skin color features of the hand region of the hand image to be repaired. The hand region in the hand image to be repaired is occluded to obtain an occlusion map of the hand image to be repaired; or, the hand region in the hand image to be repaired is replaced with the target 3D reconstruction map to obtain an occlusion map of the hand image to be repaired. The occlusion map of the hand image to be repaired is input into a reference encoder to obtain the image background features of the hand image to be repaired.

7. The method according to claim 1 or 6, characterized in that, The image generation model includes a reference Unet and a denoising Unet. The reference Unet and the denoising Unet have the same number of layers. The features output by each layer of the reference Unet are input to the corresponding layer of the denoising Unet. The step of inputting the two-dimensional structural features of the hand, the three-dimensional structural features of the hand, the skin color features of the hand, and the background features of the image into an image generation model to obtain a repaired hand image includes: The image background features are input into the reference Unet, the hand skin color features are input into each layer of the reference Unet and the denoising Unet, and the hand two-dimensional structure features and the hand three-dimensional structure features are input into the denoising Unet. The repaired hand image is generated by obtaining and decoding the features output by the denoised Unet.

8. The method according to any one of claims 1-7, characterized in that, The training process of the image generation model includes: Obtain the structural image pairs of the real hand images to be trained; Extract the two-dimensional and three-dimensional structural features of the hand to be trained from the structure map pair of the real hand image to be trained; extract the skin color features of the hand region of the real hand image to be trained; and extract the background features of the real hand image to be trained. The two-dimensional structural features of the hand to be trained, the three-dimensional structural features of the hand to be trained, the skin color features of the hand to be trained, and the background features of the image to be trained are input into the image generation model to obtain the target hand image; The image generation model is trained based on the loss value between the target hand image and the real hand image to be trained.

9. The method according to claim 8, characterized in that, The process of extracting the two-dimensional structural features of the hand from the structural map pair of the real hand image to be trained and the three-dimensional structural features of the hand from the three-dimensional reconstruction image to be trained, extracting the skin color features of the hand region of the real hand image to be trained, and extracting the background features of the real hand image to be trained, includes: The structural map of the real hand image to be trained is aligned with the two-dimensional key point map and input into the two-dimensional structure extraction model to obtain the two-dimensional structural features of the hand to be trained. The structural map of the real hand image to be trained is aligned with the 3D reconstruction map and input into the 3D structure extraction model to obtain the 3D structural features of the hand to be trained. The target hand region is segmented from the real hand image to be trained, and the target hand region is input into the CLIP model to obtain the skin color features of the hand region of the real hand image to be trained. Occlude any region in the real hand image to be trained to obtain an occlusion map of the real hand image to be trained; input the occlusion map of the real hand image to be trained into a reference encoder to obtain the background features of the hand image to be repaired.

10. The method according to claim 8, characterized in that, The image generation model includes a reference Unet and a denoising Unet. The reference Unet and the denoising Unet have the same number of layers. The features output by each layer of the reference Unet are input to the corresponding layer of the denoising Unet. The step of inputting the two-dimensional structural features of the hand to be trained, the three-dimensional structural features of the hand to be trained, the skin color features of the hand to be trained, and the background features of the image to be trained into the image generation model to obtain the target hand image includes: The background features of the image to be trained are input into the reference Unet, the skin color features of the hand to be trained are input into each layer of the reference Unet and the denoising Unet, and the two-dimensional structure features of the hand to be trained and the three-dimensional structure features of the hand to be trained are input into the denoising Unet. The target hand image is generated by obtaining and decoding the features output by the denoised Unet. The step of training the image generation model based on the loss value between the target hand image and the real hand image to be trained includes: Based on the loss value between the target hand image and the real hand image to be trained, the model parameters of the reference Une, the denoising Unet, the two-dimensional structure extraction model, and the three-dimensional structure extraction model are adjusted.

11. The method according to claim 1, characterized in that, The method further includes: The source image is input into the image confidence model to obtain confidence parameters; If the confidence parameter is less than the threshold, the process of extracting the hand image to be repaired from the source image and subsequent steps is performed.

12. A hand image restoration device, characterized in that, The device includes: The cropping unit is used to crop out the hand image to be repaired from the source image; The first acquisition unit is used to acquire a pair of structural images of the hand image to be repaired, the pair of structural images including a two-dimensional key point image and a three-dimensional reconstruction image; The retrieval unit is used to retrieve, in the hand image retrieval database, the target structural image pair with the highest similarity to the structural image pair of the hand to be repaired; The first extraction unit is used to extract the two-dimensional and three-dimensional structural features of the hand from the target structure image pair, extract the skin color features of the hand region of the hand image to be repaired, and extract the image background features of the hand image to be repaired. The repair unit is used to input the two-dimensional structural features of the hand, the three-dimensional structural features of the hand, the skin color features of the hand, and the background features of the image into the image generation model to obtain the repaired hand image.

13. A hand image restoration device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the hand image restoration method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the hand image restoration method as described in any one of claims 1-11.

Images