Hand-object interaction image generation method, system, device and storage medium

Abstract

The present invention discloses a method, system, device and storage medium for generating hand-object interaction images. The method uses the representation of model perception as a condition to explicitly model the complex self-occlusion and mutual occlusion between the hand and the object, obtains a preliminary hand image (rough image) and a hand-object topology map corresponding to the target posture, and then considers the appearance difference between the hand and the object to generate corresponding images in a divide-and-conquer manner, and then fuse them into the final hand-object interaction image. The above method can enhance the hand-object interaction image generation effect and has good application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of hand-object interaction image generation, and in particular to a hand-object interaction image generation method, system, device and storage medium. Background Art

[0002] Pose-guided image synthesis is a conditional task that aims to generate images conditioned on a target pose while preserving the identity information in the source image. This task primarily focuses on the pose transfer problem of single objects, primarily rigid objects (including humans, faces, and hands), and can be used in a variety of scenarios, such as image animation, facial reproduction, and sign language generation. In recent work, Ren et al. used a differentiable global-local attention module to generate images of people in target poses in a multi-scale manner. Deng et al. utilized a parametric 3DMM face model to decouple facial pose expressions and use them to guide face generation. Hu et al. attempted to improve the performance of gesture transfer by incorporating hand shape priors. Meanwhile, most current work on hand-object interaction focuses on simultaneously estimating the hand-object pose aligned with a given image. To better characterize the state of hand-object interaction, existing methods resort to dense triangular meshes with predefined topology generated by the hand model MANO and the modeled object. Hasson et al. leveraged physical constraints to better estimate the hand-object mesh. Cao et al. proposed an optimization-based method that leverages 2D image cues and 3D interaction priors to improve performance. Liu et al. used semi-supervised learning to utilize a large amount of external hand-object videos to help improve the performance of hand-object interaction estimation; however, the above scheme is only designed for the posture transfer problem under a single object and cannot adapt to the challenges brought by complex interaction relationships.

[0003] In addition, the effectiveness of generative adversarial networks (GANs) has been verified in the generation of realistic images of human bodies, faces, and hands. These GAN-based synthesis methods can use different input information as conditions, such as simple sketches, 2D sparse key points, and dense semantic masks. GestureGAN solves the generation of isolated hands. It uses 2D sparse hand key points as conditions and attempts to generate target hand images based on the optical flow learned from the source and target. However, these research works do not consider the generation problem of two interacting instances, and the methods used cannot adapt to the challenges brought about by the generation of interacting instances. Specifically, they do not consider 3D information or the occlusion relationship between the hand and the object, resulting in weak layering and poor results in the generated images. Summary of the Invention

[0004] The purpose of the present invention is to provide a method, system, device and storage medium for generating hand-object interaction images, which can take into account the complex occlusion between the hand and the object and generate a hand-object target image using a divide-and-conquer approach.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for generating a hand-object interaction image, comprising:

[0007] Data acquisition phase: Acquire source images, source poses, object information, and target poses. Both the source poses and target poses contain hand models and object models, as well as information about the interaction poses between the hand and object models.

[0008] The occlusion-aware topological modeling stage: using the source posture to map the hand image in the source image to the pre-constructed unified space, and obtain the conversion flow from the source image space to the unified space; locating the visual texture of the hand by calculating the occluded part of the hand in the source image, and combining the conversion flow from the source image space to the unified space to obtain the texture image; using the target posture to calculate the conversion flow from the unified space to the hand-object interaction image space and combining the texture image, obtain a preliminary hand image corresponding to the target posture; mapping the object texture contained in the object information to the unified space, and combining the conversion flow from the unified space to the hand-object interaction image space to obtain a preliminary object image corresponding to the target posture; and generating a hand topology map and an object topology map corresponding to the target posture in combination with the hand-object interaction image plane;

[0009] Hand-object interaction image generation stage: crop the source image through the hand-object foreground mask to obtain an image excluding the hand and the object, and generate a background image by filling; use the hand image and hand topology map corresponding to the preliminary target posture to generate a hand image corresponding to the target posture; use the object image and object topology map corresponding to the preliminary target posture to generate an object image corresponding to the target posture; fuse the background image, the hand image corresponding to the target posture model, and the object image corresponding to the target posture to generate a hand-object interaction image.

[0010] A hand-object interaction image generation system, comprising:

[0011] A data acquisition unit, used in the data acquisition stage, includes: acquiring a source image, a source posture, object information, and a target posture; wherein the source posture and the target posture both include a hand model and an object model, as well as interaction posture information between the hand model and the object model;

[0012] The occlusion-aware topological modeling unit is applied to the occlusion-aware topological modeling stage, including: using the source posture to map the hand image in the source image to a pre-constructed unified space to obtain a conversion flow from the source image space to the unified space; locating the visual texture of the hand by calculating the occluded part of the hand in the source image, and obtaining a texture image in combination with the conversion flow from the source image space to the unified space; using the target posture to calculate the conversion flow from the unified space to the hand-object interaction image space and combining the texture image to obtain a preliminary hand image corresponding to the target posture; mapping the object texture contained in the object information to the unified space, and combining the conversion flow from the unified space to the hand-object interaction image space to obtain a preliminary object image corresponding to the target posture; and generating a hand topology map and an object topology map corresponding to the target posture in combination with the hand-object interaction image plane;

[0013] The hand-object generator is applied to the hand-object interaction image generation stage, including: cropping the source image through the hand-object foreground mask to obtain an image excluding the hand and the object, and generating a background image by filling; using the hand image and hand topology map corresponding to the preliminary target posture to generate a hand image corresponding to the target posture; using the object image and object topology map corresponding to the preliminary target posture to generate an object image corresponding to the target posture; fusing the background image, the hand image corresponding to the target posture model and the object image corresponding to the target posture to generate a hand-object interaction image.

[0014] A processing device comprising: one or more processors; a memory for storing one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0016] A readable storage medium stores a computer program, which implements the aforementioned method when the computer program is executed by a processor.

[0017] It can be seen from the technical solution provided by the present invention that, by using the representation of model perception as a condition, the complex self-occlusion and mutual occlusion between the hand and the object are explicitly modeled to obtain a preliminary hand image (rough image) and hand-object topology map corresponding to the target posture. Then, considering the appearance differences between the hand and the object, the corresponding images are generated separately in a divide-and-conquer manner, and then fused into the final hand-object interaction image. The above method can enhance the hand-object interaction image generation effect and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart of a method for generating a hand-object interaction image provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram illustrating the definition of a hand-object interaction image generation task provided by an embodiment of the present invention;

[0021] Figure 3 This is an overall framework diagram of a method for generating a hand-object interaction image provided by an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of a hand-object interaction image generation system provided by an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of a processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0025] First, the following terms may be used in this article:

[0026] The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y includes both “X” or “Y” and “X and Y”.

[0027] The terms "include," "comprises," "contains," "has," or other similar expressions should be interpreted as non-exclusive. For example, "including certain technical features (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products, or manufactured articles, etc.) should be interpreted as including not only the technical features explicitly listed, but also other technical features known in the art that are not explicitly listed.

[0028] The following describes in detail the method, system, device, and storage medium for generating hand-object interaction images provided by the present invention. Any information not described in detail in the embodiments of the present invention is prior art known to those skilled in the art. For any unspecified conditions in the embodiments of the present invention, the conditions conventional in the art or recommended by the manufacturer shall prevail.

[0029] Example 1

[0030] An embodiment of the present invention provides a method for generating hand-object interaction images, which is used to complete a new hand-object interaction image generation task. It has rich application prospects and can be used in scenarios such as AR / VR (augmented reality / virtual reality) games and online shopping. For example, when a consumer shops online, interactive visualization will give him / her an immersive experience. In addition, if a consumer wants to add a name to an object (for example, on a mobile phone), this goal can be achieved through object texture editing using some corresponding applications in the occlusion-aware topological modeling of the present invention. In addition, in online shopping scenarios, consumers usually do not have the corresponding products. They only need to upload a photo of their hand to generate a hand-object image that retains the hand's identity, presenting a realistic interactive experience. The task completed by the present invention can be used to synthesize hand-object interaction images, thereby improving the performance of other subsequent tasks. For example, current methods for estimating hand-object interaction postures are generally based on deep learning, but their performance is limited by the size of training data due to the annotation cost. The present invention can generate hand-object interaction images that can well simulate real-world features and improve the performance of hand-object posture estimation.

[0031] The hand-object interaction image generation task in the embodiments of the present invention is a conditional generation task. Its goal is to generate a hand-object interaction image conditioned on a target pose while preserving the identity of the source image. Specifically, the object must be well-modeled and have various textures known. Solving this task is challenging because understanding the complex interactions between hands during image generation is not easy. The main challenges are summarized as follows. First, the occlusion relationship between the two interacting instances (i.e., the hand and the object) must be modeled. In hand-object interaction scenarios, complex self-occlusion and mutual occlusion often occur. Since occlusion increases the complexity of the transition between sources, these occluded regions should be located and identified, which will facilitate the final generation. Second, the different characteristics between the two instances need to be taken into account during the generation process. Specifically, the hand has joints and self-occlusion occurs between the joints, while the object is usually a rigid body with fine textures. Therefore, the generated hand-object interaction image should have interaction instances with realistic appearance and reasonable interaction between the two.

[0032] To this end, the present invention proposes a framework (called the HOGAN framework) to meet the challenges of this new task. The HOGAN framework includes two parts: occlusion-aware topological modeling and hand-object generation. The occlusion-aware topological modeling part uses the model-aware representation as a condition and uses its inherent structural topology to construct a unified space. In the unified space, the complex self-occlusion and mutual occlusion between the hand and the object are explicitly considered. Specifically, the visible parts of the hand and object and their corresponding fine-grained topological maps are mapped to the target image plane. At the same time, using the unified space as an intermediate variable for constructing the transformation flow, the transformation flow between the source and the target can be directly calculated. The results obtained in this part provide rich information for the final image synthesis. In the hand-object generation part, the appearance difference between the hand and the object is considered and the final hand-object interaction image is generated in a divide-and-conquer manner. As Figure 1 FIG. 1 shows the main process of a method for generating a hand-object interaction image provided by an embodiment of the present invention, which includes the following steps:

[0033] Step 1: Data acquisition phase.

[0034] In the embodiment of the present invention, the source image, source posture, object information and target posture are mainly obtained, wherein the source posture and target posture are both hand-object interaction postures.

[0035] Figure 2 The definition of the hand-object interaction image generation task in an embodiment of the present invention is presented, which aims to generate a hand-object interaction image in a target posture while maintaining the appearance of the source image. Figure 2 Each row in the figure represents an example of using the acquired information to generate a hand-object interaction image. From left to right, each column represents: source image (Source Image), object information (ObjectInfo.), source posture (Source Posture), target posture (Target Posture), and generated hand-object interaction image.

[0036] In this embodiment of the present invention, both the source and target poses contain a hand model, an object model, and information about their interaction pose. The source pose is aligned with the source image, and the target pose is aligned with the desired hand-object interaction image. Object information primarily includes the object model (Obj.Model) of the object's texture.

[0037] Step 2: Occlusion-aware topology modeling phase.

[0038] In this stage: the source posture is used to map the hand image in the source image to the pre-constructed unified space to obtain the conversion flow from the source image space to the unified space, the visual texture of the hand is located by calculating the occluded part of the hand in the source image, and the texture image is obtained by combining the conversion flow from the source image space to the unified space, the target posture is used to calculate the conversion flow from the unified space to the hand-object interaction image space and combined with the texture image to obtain a preliminary hand image corresponding to the target posture; the object texture contained in the object information is mapped to the unified space, and combined with the conversion flow from the unified space to the hand-object interaction image space to obtain a preliminary object image corresponding to the target posture; and the hand topology map and object topology map corresponding to the target posture are generated in combination with the hand-object interaction image plane. Specifically, the preferred implementation methods of this stage are as follows:

[0039] All kinds of information obtained in this stage belong to the representation information of the hand object, and the representation information is generated by the model and has an inherent topological structure (a group of patches composed of many nodes, and the connection relationship between the nodes is fixed). Therefore, it is called model-aware representation. Here we first give an overview of the model-aware representation used. The hands in the source pose and the target pose can be represented by two sets of MANO models, and the objects can be represented by two sets of YCB models. Both MANO and YCB provide triangular mesh expressions, which can densely depict the structure of the hand and the object. Specifically, the hand model representation contains N v vertices and N f triangular faces (referred to as faces), whose mesh is represented by According to the model information, fixed topology information and inherent topology information can be determined It is organized as a vertex triplet, where each unit is recorded as the corresponding vertex coordinate aligned with the plane. In the following description, s, t, and u refer to the symbols of the source image space, the target space (hand-object interaction image space), and the unified space, respectively.

[0040] First, based on inherent topological information, the surfaces of the hand and object models are disentangled to construct a unified space. In this unified space, the same representation is bound to the same mesh face, regardless of pose state. This means that the same facet in the hand model (or object model) in both the source and target poses is mapped to the same representation in the unified space. This unified space enables source-to-target mapping and can interpolate pre-known object textures.

[0041] Afterwards, the hand image in the source image is mapped to the unified space using the source posture in an occlusion-aware manner (occlusion-based perceptual modeling) to obtain the transformation flow T from the source image space to the unified space. u←s .

[0042] Among them, the conversion flow T from the source image space to the unified space at the position (x, y) in the unified space is u←s(x,y) is represented as:

[0043] T u←s (x,y)=W u (x,y)·P s (F u (x,y))

[0044] Among them, x represents the horizontal axis position, y represents the vertical axis position, F u (x,y) represents the face index at the (x,y) position in the uniform space, P s (F u (x, y)) represents the model of the source pose (including the hand model and the object model, which are determined by the position) and the face index F u (x, y) corresponds to the coordinates of the three vertices of the face, W u (x,y) represents the relative weight of the face at the (x,y) position in the unified space, which can be determined based on the pose and the alignment relationship of the corresponding image space.

[0045] Furthermore, occlusion (including self-occlusion and mutual occlusion) is calculated synchronously, expressed as:

[0046] O u←s (x,y)=(F u (x,y)≠F s (T u←s (x,y)))

[0047] Among them, O u←s (x,y) indicates that the position (x,y) is blocked, F s (T u←s (x,y)) represents the transformation T u←s (x,y) positioned source image F s The image at the corresponding position in F u (x,y)≠F s (T u←s (x,y)) refers to F s (T u←s The image corresponding to (x,y)) has no corresponding face index in the unified space, so the position (x,y) is considered to be blocked;

[0048] From this, we can locate the visual texture of the hand: 1-O u←s (x,y).

[0049] Combine the transformation flow T from the source image space to the unified space u←s Map the visual texture of the hand to the unified space to obtain the initial texture image, which is expressed as:

[0050] I u =Warp(T u←s,I s )⊙(1-O u←s )

[0051] Among them, I u represents the initial texture image, which is aligned with the uniform space; ⊙ and Warp(·) represent the element-wise multiplication and mapping operations, respectively.

[0052] Since the objects in the source image inevitably contain occluded areas, this is not sufficient for target generation. Therefore, the texture in the initial texture image is replaced with the pre-stored hand texture to obtain the final texture image.

[0053] Then use the target posture to calculate the conversion flow T from the unified space to the hand-object interaction image space t←u , where the transformation flow T from the unified space to the hand-object interaction image space at the position (x′, y′) in the hand-object interaction image space is t←u (x′,y′) is expressed as:

[0054] T t←u (x′,y′)=W t (x′,y′)·P u (F t (x′,y′))

[0055] Among them, x′ represents the horizontal axis position, y′ represents the vertical axis position, and F t (x′, y′) represents the face index of the position (x′, y′) in the hand-object interaction image space, P u (F t (x′, y′)) represents the model of the unified space (including the hand model and the object model, which are determined by the position) and the face index F t (x, y) corresponds to the coordinates of the three vertices of the face, W t (x′, y′) represents the relative weight of the face at the position (x′, y′) in the hand-object interaction image space.

[0056] Since the hand-object interaction image space and the target posture space are aligned, F t Can be determined based on the target pose.

[0057] Finally, the transformation flow T from the unified space to the hand-object interaction image space is t←u Under the guidance of Get the hand image I corresponding to the preliminary target posture t , expressed as:

[0058]

[0059] Wherein, Warp(·) represents a mapping operation.

[0060] In addition, since object information can provide rough object texture, after mapping the object texture to the unified space, the conversion flow from the unified space to the hand-object interaction image space is combined to obtain the preliminary object image corresponding to the target posture.

[0061] At the same time, in order to provide sufficient guidance for the next stage, it is necessary to generate a fine-grained topology map Y t , the topology graph at position (x′, y′) is generated synchronously as follows:

[0062] Y t (x′,y′)=Bary(P u (F t (x′,y′)))

[0063] Bary(·) refers to the center of gravity of the corresponding surface in the surface space. The above formula is combined with the position area of ​​the hand and the position area of ​​the object under the target posture to obtain the corresponding hand topology map and object topology map.

[0064] Step 3: Hand-object interaction image generation stage.

[0065] In this stage: the source image is cropped using the hand-object foreground mask to obtain an image that does not contain the hand and the object, and a background image is generated by filling; the hand image corresponding to the target posture is generated using the preliminary hand image and hand topology map corresponding to the target posture; the object image corresponding to the target posture is generated using the preliminary object image and object topology map corresponding to the target posture; the background image, the hand image corresponding to the target posture model, and the object image corresponding to the target posture are fused to generate a hand-object interaction image. Specifically, the preferred implementation methods of this stage are as follows:

[0066] Considering the different properties of hands and objects, a hand-object generator is designed in the present invention to generate a hand-object interaction image (target image) in a divide-and-conquer manner. It consists of three branches: background branch, object branch, and hand branch.

[0067] 1) The background branch has a first generative network responsible for generating the inpainted background image. Specifically, it uses the hand-object foreground mask (background), which can be obtained by projecting the source pose onto the 2D plane, to fill in the background cropped from the source image.

[0068] 2) The object branch has a second generative network responsible for generating the object image (Obj.Foreground). The preliminary object image corresponding to the target pose (Obj.input) is used as the input of the second generative network. To enable the object branch to perceive the object's structural information, the object topology (Obj.Topo.) is injected into the second generative network using spatially adaptive normalization (SPADE). The second generative network then generates the object image corresponding to the target pose.

[0069] 3) The hand branch has a third generative network responsible for generating a hand image (HandForeground) corresponding to the target pose model. The preliminary hand image corresponding to the target pose (Hand input) is used as the input to the third generative network. Simultaneously, the hand topology (Hand Topo.) is injected into the third generative network using spatially adaptive normalization. The third generative network then generates a hand image corresponding to the target pose.

[0070] Exemplarily, the above three generating networks may use a U-shaped structure network (Unet network). The specific network structure and principle may refer to conventional technology, and will not be elaborated in the present invention.

[0071] At the same time, during training, a part of the source poses is set to be equal to the corresponding target poses. Then, according to the topological modeling stage of occlusion perception and the hand-object interaction image generation stage, the generated hand-object interaction image is the reconstructed source image, such as Figure 3 As shown in the upper part, they are now integrated into the generation process through an attention sampler.

[0072] In the embodiment of the present invention, the three branches process three instances with different attributes, namely background, object and hand, respectively, and the results of the three branches are merged through the fusion module (Fusion). By extracting the last layer features before the output layer of the three generative networks (the output layer refers to the layer that outputs the image), the present invention uses two convolutional layers to learn a fusion mask respectively, and obtains two fusion masks, namely the hand mask M. h and hand-object mask M f , respectively refer to the unoccluded hand and hand-object foreground. The fusion module merges the results of the three branches into the final generated result, which is expressed as:

[0073] I=(I h ⊙M h +I o ⊙(1-M h ))⊙M f +I b ⊙(1-M f )

[0074] Among them, I represents the hand-object interaction image, I hrepresents the hand image corresponding to the target posture model, I o Represents the object image, I b represents the background image, and ⊙ represents the element-wise multiplication operation.

[0075] On the other hand, the above-mentioned hand-object generator is trained; the overall loss of training consists of three parts.

[0076] The first part is the perceptual loss on the generated hand-object interaction image, expressed as:

[0077]

[0078] Among them, f i (·) refers to the feature extractor of layer i, x t and They represent the real hand-object interaction image (known image) and the generated hand-object interaction image respectively.

[0079] In an embodiment of the present invention, this partial perceptual loss uses a newly introduced pre-trained network. For example, the 2nd, 7th, 12th, 21st, and 30th layer feature extractors of the pre-trained VGG network can be used to extract relevant image features.

[0080] The second part is the reconstruction loss on the source image. The reconstruction loss is calculated when the target pose and the source pose are the same. Specifically, the generated hand-object interaction image is the reconstructed source image. The reconstruction loss on the source image is calculated using the reconstructed source image and the acquired source image, which is expressed as:

[0081]

[0082] Among them, x s and represent the acquired source image and the reconstructed source image respectively.

[0083] The third component is the adversarial loss, which is used to constrain the distribution of generated hand-object interaction images and real hand-object interaction images. Specifically, this paper designs a discriminator and trains it using adversarial learning to improve the visual performance of the generated hand-object interaction images. The discriminator is defined as D(·), and the adversarial loss is:

[0084]

[0085]

[0086] Among them, x t and They represent the real hand-object interaction image (known image) and the generated hand-object interaction image, respectively. c represents the combination of the generated object topology map and the hand topology map. represents the adversarial loss of the hand-object generator, is the adversarial loss of the discriminator, represents the mathematical expectation, D(x t |c)) represents the probability that the discriminator judges the input image as a generated image or a real image, given the combination c of the generated object topology map and the hand topology map.

[0087] The final overall loss is:

[0088]

[0089] Among them, λ1 and λ2 are weight factors used to balance the related loss functions.

[0090] Figure 3 The overall framework of the present invention is shown. The dotted box in the lower left corner represents the occlusion-aware topology modeling stage (Occlusion-Aware Topology Modeling), which shows the process of mapping the 3D hand-object model to the 2D plane, expressing how the source pose and target pose are associated with the unified space, and the output that can be obtained through the association, including the rendered image of the hand and object, and the object topology map of the hand and object. The dotted box in the lower right corner represents the hand-object interaction image generation stage (Hand-Object Generator), which shows the processing flow of the three branches; the upper part shows the overall training framework, with four input information on the left (that is, the four information mentioned in the aforementioned data acquisition stage), followed by the occlusion-aware topology modeling stage (abbreviated as Topology Modeling), and the hand-object interaction image is generated through the hand-object interaction image generation stage (abbreviated as HO-Gen.). In the training framework, some source poses will be set equal to the corresponding target poses. At this time, the hand-object interaction image generated is the reconstructed source image (Reconstructed Source Image). In other cases, the source pose is different from the corresponding target pose, and the generated hand-object interaction image is the generated target image (Generated Target Image).

[0091] Through the description of the above embodiments, those skilled in the art will clearly understand that the above embodiments can be implemented through software or by using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) and includes a number of instructions for causing a computer device (such as a personal computer, a server, or a network device) to execute the methods described in the various embodiments of the present invention.

[0092] Example 2

[0093] The present invention also provides a hand-object interaction image generation system, which is mainly implemented based on the method provided in the above embodiment, such as Figure 4 As shown, the system mainly includes:

[0094] A data acquisition unit, used in the data acquisition stage, includes: acquiring a source image, a source posture, object information, and a target posture; wherein the source posture and the target posture both include a hand model and an object model, as well as interaction posture information between the hand model and the object model;

[0095] The occlusion-aware topological modeling unit is applied to the occlusion-aware topological modeling stage, including: using the source posture to map the hand image in the source image to a pre-constructed unified space to obtain a conversion flow from the source image space to the unified space; locating the visual texture of the hand by calculating the occluded part of the hand in the source image, and obtaining a texture image in combination with the conversion flow from the source image space to the unified space; using the target posture to calculate the conversion flow from the unified space to the hand-object interaction image space and combining the texture image to obtain a preliminary hand image corresponding to the target posture; mapping the object texture contained in the object information to the unified space, and combining the conversion flow from the unified space to the hand-object interaction image space to obtain a preliminary object image corresponding to the target posture; and generating a hand topology map and an object topology map corresponding to the target posture in combination with the hand-object interaction image plane;

[0096] The hand-object generator is applied to the hand-object interaction image generation stage, including: cropping the source image through the hand-object foreground mask to obtain an image excluding the hand and the object, and generating a background image by filling; using the hand image and hand topology map corresponding to the preliminary target posture to generate a hand image corresponding to the target posture; using the object image and object topology map corresponding to the preliminary target posture to generate an object image corresponding to the target posture; fusing the background image, the hand image corresponding to the target posture model and the object image corresponding to the target posture to generate a hand-object interaction image.

[0097] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0098] Example 3

[0099] The present invention also provides a processing device, such as Figure 5As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the aforementioned embodiment.

[0100] Furthermore, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0101] In the embodiment of the present invention, the specific types of the memory, input device, and output device are not limited; for example:

[0102] The input device can be a touch screen, image acquisition device, physical button or mouse;

[0103] The output device may be a display terminal;

[0104] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0105] Example 4

[0106] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0107] In the embodiments of the present invention, the computer-readable storage medium may be provided in the aforementioned processing device, for example, as a memory in the processing device. Alternatively, the computer-readable storage medium may be a USB flash drive, a removable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0108] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for generating a hand-object interaction image, characterized in that: include: Data acquisition phase: Acquire source images, source poses, object information, and target poses. Both the source poses and target poses contain hand models and object models, as well as information about the interaction poses between the hand and object models. The occlusion-aware topological modeling stage: using the source posture to map the hand image in the source image to the pre-constructed unified space, and obtain the conversion flow from the source image space to the unified space; locating the visual texture of the hand by calculating the occluded part of the hand in the source image, and combining the conversion flow from the source image space to the unified space to obtain the texture image; using the target posture to calculate the conversion flow from the unified space to the hand-object interaction image space and combining the texture image, obtain a preliminary hand image corresponding to the target posture; mapping the object texture contained in the object information to the unified space, and combining the conversion flow from the unified space to the hand-object interaction image space to obtain a preliminary object image corresponding to the target posture; and generating a hand topology map and an object topology map corresponding to the target posture in combination with the hand-object interaction image plane; Hand-object interaction image generation stage: crop the source image using the hand-object foreground mask to obtain an image excluding the hand and object, and generate a background image by filling; generate a hand image corresponding to the target posture using the preliminary hand image and hand topology map corresponding to the target posture; generate an object image corresponding to the target posture using the preliminary object image and object topology map corresponding to the target posture; fuse the background image, the hand image corresponding to the target posture model, and the object image corresponding to the target posture to generate a hand-object interaction image; The method of locating the visual texture of the hand by calculating the occluded portion of the hand in the source image and obtaining the texture image by combining the conversion flow from the source image space to the unified space includes: Calculate the occlusion position, expressed as: O u←s (x,y)=(F u (x,y)≠F s (T u←s (x,y))) Among them, x represents the horizontal axis position, y represents the vertical axis position, u←s (x,y) indicates that the position (x,y) is blocked, F u (x,y) represents the face index at the (x,y) position in the uniform space, T u←s (x,y) represents the transformation flow from the source image space to the uniform space at the position (x,y), F s (T u←s (x,y)) represents the transformation flow T u←s (x,y) positioned source image F s The image at the corresponding position in F u (x,y)≠F s (T u←s (x,y)) refers to F s (T u←s The image corresponding to (x,y)) has no corresponding face index in the uniform space; Locate the visual texture of the hand: 1-0 u←s (x,y); Combine the transformation flow T from the source image space to the unified space u←s The visual texture of the hand O u←s Mapped to the uniform space, the initial texture image is obtained, which is expressed as: I u =Warp(T u←s ,I s )⊙(1-O u←s ) Among them, I u represents the initial texture image, ⊙ and Warp(·) represent the element-wise multiplication and mapping operations, respectively; Use the pre-stored hand texture to replace the texture in the initial texture image to obtain the final texture image 2. The method for generating a hand-object interaction image according to claim 1, characterized in that: The unified space is constructed by untangling the surfaces of the hand model and the object model, and mapping the hand image in the source image to the unified space using the source posture to obtain the transformation flow T from the source image space to the unified space. u←s ; Among them, at the position (x, y) in the unified space, the conversion flow T from the source image space to the unified space is u←s (x,y) is represented as: T u←s (x,y)=W u (x,y)·P s (F u (x,y)) Among them, x represents the horizontal axis position, y represents the vertical axis position, F u (x,y) represents the face index at the (x,y) position in the uniform space, P s (F u (x,y)) represents the face index F in the model of the source pose u (x, y) corresponds to the coordinates of the three vertices of the face, W u (x,y) represents the relative weight of the face at the (x,y) position in uniform space.

3. The method for generating a hand-object interaction image according to claim 1, wherein: The step of calculating a conversion flow from the unified space to the hand-object interaction image space using the target posture and combining the texture image to obtain a preliminary hand image corresponding to the target posture includes: The target pose is used to calculate the transformation flow T from the unified space to the hand-object interaction image space. t←u , where the conversion flow T from the unified space to the hand-object interaction image at the position (x′, y′) in the hand-object interaction image space is t←u (x′,y′) is expressed as: T t←u (x′,y′)=W t (x′,y′)·P u (F t (x′,y′)) Among them, x′ represents the horizontal axis position, y′ represents the vertical axis position, and F t (x, y) represents the face index of the position (x′, y′) in the hand-object interaction image space, P u (F t (x′, y′)) represents the face index F in the unified space model t The coordinates of the three vertices of the face corresponding to (x′, y′), W t (x′, y′) represents the relative weight of the face at position (x′, y′) on the hand-object interaction image; The conversion flow T from the unified space to the hand-object interaction image t←u Under the guidance of Get the hand image I corresponding to the preliminary target posture t , expressed as: Wherein, Warp(·) represents a mapping operation.

4. The method for generating a hand-object interaction image according to claim 1, wherein: The step of generating a hand topology map and an object topology map corresponding to a target posture by combining the hand-object interaction image plane includes: The topology graph at position (x′, y′) is generated synchronously as follows: AND t (x′,y′)=Barty(P u (F t (x′,y′))) Where Bary(·) refers to the center of gravity of the corresponding surface in the surface space, (x′, y′) is the position in the hand-object interaction image space, x′ represents the horizontal axis position, y′ represents the vertical axis position, and F t (x, y) represents the face index of the position (x′, y′) in the hand-object interaction image space, P u (F t (x′, y′)) represents the face index F in the unified space model t The coordinates of the three vertices of the face corresponding to (x′, y′); The above formula is combined with the hand position area and object position area under the target posture to obtain the corresponding hand topology map and object topology map.

5. The method for generating a hand-object interaction image according to claim 1, wherein: The hand-object interaction image generation stage is implemented using a hand-object generator, and the background branch is provided with a first generation network responsible for generating the background image; The object branch is equipped with a second generative network. The preliminary object image corresponding to the target pose is used as the input of the second generative network. At the same time, the object topology map is injected into the second generative network using a spatially adaptive normalization method to generate the object image corresponding to the target pose. The hand branch is equipped with a third generative network. The preliminary hand image corresponding to the target posture is used as the input of the third generative network. At the same time, the hand topology map is injected into the third generative network using a spatial adaptive normalization method to generate a hand image corresponding to the target posture. By extracting the last layer of features before the output layer of the three generative networks and using two convolutional layers to learn a fusion mask respectively, two fusion masks are obtained, which are called hand masks M. h and hand-object mask M f , respectively refer to the unobstructed hand and hand-object foreground, and the fusion method is expressed as: I=(I h ⊙M h +I o ⊙(1-M h ))⊙M f +I b ⊙(1-M f ) Among them, I represents the hand-object interaction image, I h represents the hand image corresponding to the target posture model, I o Represents the object image, I b represents the background image, and ⊙ represents the element-wise multiplication operation.

6. The method for generating a hand-object interaction image according to claim 1 or 5, characterized in that: The hand-object interaction image generation stage is implemented using a hand-object generator, and the hand-object generator is trained; During training, some source poses are set equal to the corresponding target poses. Then, according to the occlusion-aware topology modeling stage and the hand-object interaction image generation stage, the generated hand-object interaction image is the reconstructed source image. The overall training loss consists of three parts: The first part is the perceptual loss on the generated hand-object interaction image; The second part is the reconstruction loss on the source image, which is calculated when the target posture is the same as the source posture. The generated hand-object interaction image is the reconstructed source image. The reconstruction loss on the source image is calculated using the reconstructed source image and the acquired source image. The third part is the adversarial loss, which is used to constrain the distribution of generated hand-object interaction images and real hand-object interaction images.

7. A hand-object interaction image generation system, characterized in that: The method according to any one of claims 1 to 6 is implemented, and the system comprises: A data acquisition unit, used in the data acquisition stage, includes: acquiring a source image, a source posture, object information, and a target posture; wherein the source posture and the target posture both include a hand model and an object model, as well as interaction posture information between the hand model and the object model; The occlusion-aware topological modeling unit is applied to the occlusion-aware topological modeling stage, including: using the source posture to map the hand image in the source image to a pre-constructed unified space to obtain a conversion flow from the source image space to the unified space; locating the visual texture of the hand by calculating the occluded part of the hand in the source image, and obtaining a texture image in combination with the conversion flow from the source image space to the unified space; using the target posture to calculate the conversion flow from the unified space to the hand-object interaction image space and combining the texture image to obtain a preliminary hand image corresponding to the target posture; mapping the object texture contained in the object information to the unified space, and combining the conversion flow from the unified space to the hand-object interaction image space to obtain a preliminary object image corresponding to the target posture; and generating a hand topology map and an object topology map corresponding to the target posture in combination with the hand-object interaction image plane; The hand-object generator is applied to the hand-object interaction image generation stage, including: cropping the source image through the hand-object foreground mask to obtain an image excluding the hand and the object, and generating a background image by filling; using the hand image and hand topology map corresponding to the preliminary target posture to generate a hand image corresponding to the target posture; using the object image and object topology map corresponding to the preliminary target posture to generate an object image corresponding to the target posture; fusing the background image, the hand image corresponding to the target posture model and the object image corresponding to the target posture to generate a hand-object interaction image.

8. A processing device, characterized in that include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images