Interactive image generation model, interactive image generation method, device and equipment

Abstract

The invention provides an interactive image generation model, an interactive image generation method, an interactive image generation device and interactive image generation equipment, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and is applied to scenes such as AIGC (Artificial Intelligence Generation Content) and the like. An interactive image generation model is provided and comprises an object feature processing network, a structural relation processing network, a hand structural feature processing network, a hand restoration network, an object restoration network and a decoder. The interactive image generation model processes the image containing the target object to obtain appearance features of the target object; fusing the reference relation graph with the appearance features, and outputting interaction features; fusing the hand structure feature output by the hand structure feature processing network, the interaction feature and the generated hand texture feature to obtain a first feature map; fusing the first feature map and the generated object texture features to obtain a to-be-decoded second feature map; and decoding the second feature graph to generate an interaction graph.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, deep learning, and large models, and specifically to an interactive image generation model, an interactive image generation method, a device, and equipment. Background Art

[0002] The current method of generating interactive human videos is usually achieved through a diffusion model that combines single image completion with video optical flow. However, the generation process of hands and objects in this method is very likely to affect each other, and because the target object and the original object may differ in size and shape, the objects in the generated interactive image may be distorted and may produce unreasonable human interaction relationships, resulting in poor effects and quality of the generated interactive image. Summary of the invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0004] The first embodiment of the present disclosure proposes an interactive image generation model, including: an object feature processing network, a structure relationship processing network, a hand structure feature processing network, a hand restoration network, an object restoration network and a decoder;

[0005] The object feature processing network is used to process an image containing a target object to obtain appearance features of the target object;

[0006] The structural relationship processing network is used to fuse the input reference relationship graph with the appearance feature and output an interactive feature;

[0007] The hand restoration network is used to fuse the hand structure features output by the hand structure feature processing network, the interaction features and the generated hand texture features to obtain a first feature map;

[0008] The object restoration network is used to fuse the first feature map and the generated object texture features to obtain a second feature map to be decoded;

[0009] The decoder is used to decode the second feature map to generate an interaction map including the target object.

[0010] A second aspect of the present disclosure provides a method for generating an interactive image, including:

[0011] Obtain reference relationship graph, target object graph and hand structure features;

[0012] Processing the target object image to determine the appearance features of the target object;

[0013] Processing the reference relationship graph and the appearance features to determine interaction features;

[0014] The interaction feature, the preset hand texture feature and the hand structure feature are integrated to obtain a first feature map;

[0015] The first feature map and the preset object texture feature map are merged to obtain a second feature map;

[0016] Based on the second feature map, an interaction map including the target object is generated.

[0017] The third aspect of the present disclosure provides a device for generating an interactive image, including:

[0018] A first acquisition module is used to acquire a reference relationship graph, a target object graph and hand structure features;

[0019] A first determination module, used to process the target object image to determine the appearance features of the target object;

[0020] A second determination module, configured to process the reference relationship graph and the appearance feature to determine an interaction feature;

[0021] A second acquisition module, used for fusing the interaction feature, the preset hand texture feature and the hand structure feature to obtain a first feature map;

[0022] A third acquisition module, used for fusing the first feature map and a preset object texture feature map to obtain a second feature map;

[0023] A generating module is used to generate an interaction graph including the target object based on the second feature graph.

[0024] The fourth aspect embodiment of the present disclosure proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating an interactive image proposed in the second aspect embodiment of the present disclosure is implemented.

[0025] The fifth aspect embodiment of the present disclosure proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method for generating an interactive image proposed in the second aspect embodiment of the present disclosure is implemented.

[0026] The sixth aspect of the present disclosure provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method for generating an interactive image as provided in the second aspect of the present disclosure.

[0027] The interactive image generation model, interactive image generation method, device and equipment provided by the present disclosure have the following beneficial effects:

[0028] In the embodiment of the present disclosure, an interactive image generation model is proposed, including an object feature processing network, a structure relationship processing network, a hand structure feature processing network, a hand restoration network, an object restoration network and a decoder. First, the image containing the target object is processed by the object feature processing module to obtain the appearance features of the target object, and then the input reference relationship image is fused with the appearance features by the structure relationship processing network to output the interactive features. Then, the hand structure features, interactive features and the generated hand texture features output by the hand structure feature processing network are fused by the hand restoration network to obtain a first feature map, and the first feature map and the generated object texture features are fused by the object restoration map to obtain a second feature map to be decoded, and finally the second feature map is decoded by the decoder to generate an interactive map containing the target object. Therefore, the interactive image generation model proposed in the present disclosure realizes the feature decoupling of the hand and the target object in the process of generating the interactive map, improves the generation quality of the hand and the object, and makes the generated interactive map effectively avoid the problems of object distortion and unreasonable interactive relationship, which provides conditions for improving the effect and quality of the generated interactive map.

[0029] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0031] Figure 1 A schematic diagram of the structure of an interactive image generation model provided by an embodiment of the present disclosure;

[0032] Figure 2 A schematic diagram of a process for generating a reference relationship diagram provided in an embodiment of the present disclosure;

[0033] Figure 3 A schematic diagram of the specific structure of each network in the interactive image generation model provided in an embodiment of the present disclosure;

[0034] Figure 4 A schematic diagram of the training process of the interactive image generation model provided in an embodiment of the present disclosure;

[0035] Figure 5 A schematic diagram of a flow chart of a preprocessing component in an interactive image generation model provided in an embodiment of the present disclosure processing a reference relationship graph;

[0036] Figure 6 A schematic diagram of a process of processing a reference relationship graph by a preprocessing component in an interactive image generation model provided in an embodiment of the present disclosure;

[0037] Figure 7 A schematic diagram of a flow chart of a method for generating an interactive image provided by an embodiment of the present disclosure;

[0038] Figure 8 A schematic diagram of a flow chart of a method for generating an interactive image provided by an embodiment of the present disclosure;

[0039] Fig. 9 A schematic diagram of a flow chart of a method for generating an interactive image provided by an embodiment of the present disclosure;

[0040] Fig.10 A schematic diagram of a flow chart of a method for generating an interactive image provided by an embodiment of the present disclosure;

[0041] Fig.11 A schematic diagram of the structure of a device for generating an interactive image provided by an embodiment of the present disclosure;

[0042] Fig.12 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0043] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0044] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as computer vision, deep learning, and large models.

[0045] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0046] Computer vision refers to the use of cameras and computers to replace the human eye to identify, track and measure targets, and further perform graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection.

[0047] Deep Learning (DL) is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images, and sounds. The ultimate goal of deep learning is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images, and sounds.

[0048] The large model can also be called the Foundation Model. The model extracts knowledge through billions of corpora or images, learns and then produces a large model with billions of parameters.

[0049] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0050] The interactive image generation model, interactive image generation method, device and apparatus according to the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0051] Figure 1 A schematic diagram of the structure of an interactive image generation model provided in an embodiment of the present disclosure.

[0052] like Figure 1 As shown, the interactive image generation model 100 includes: an object feature processing network 101, a structure relationship processing network 102, a hand structure feature processing network 103, a hand restoration network 104, an object restoration network 105 and a decoder 106;

[0053] The object feature processing network 101 is used to process an image containing a target object to obtain appearance features of the target object;

[0054] The structural relationship processing network 102 is used to fuse the input reference relationship image with the appearance feature and output the interactive feature;

[0055] The hand restoration network 104 is used to fuse the hand structure features, interaction features and generated hand texture features output by the hand structure feature processing network 103 to obtain a first feature map;

[0056] The object restoration network 105 is used to fuse the first feature map and the generated object texture features to obtain a second feature map to be decoded;

[0057] The decoder 106 is used to decode the second feature map to generate an interaction map including the target object.

[0058] The target object may be any object that interacts with the hand, and may be determined as needed. For example, a picked-up box may be used as the target object, and the present disclosure does not limit this.

[0059] The appearance features of the target object may include features such as the shape and size of the target object, which are not limited in the present disclosure.

[0060] Among them, the reference relationship diagram can be a predetermined reference diagram containing the interaction relationship between the hand and the target object in the interaction diagram to be generated. The interaction relationship contained therein can be determined according to the needs of the actual scene. It can be used as an interaction driving signal to achieve feature decoupling of the hand and the object, which is helpful for the generation of the hand and the object.

[0061] In the present disclosure, when determining the reference relationship graph, the interaction relationship reference image can be selected according to the actual scene requirements, and the bounding boxes of the hand and the object in the reference image can be extracted respectively. Through the bounding boxes of the hand and the object, the interaction relationship between the hand and the object is obtained, thereby obtaining the reference relationship graph, such as Figure 2 As shown, Figure 2 A schematic diagram of a generation process of a reference relationship diagram provided in an embodiment of the present disclosure, wherein: Figure 2 Included in Figure 2 a and Figure 2 b are schematic diagrams of input frame images. Figure 2 c is a reference relationship diagram, Figure 2 In c, bounding box 201 and bounding box 203 are both hand bounding boxes, and bounding box 202 is the object bounding box. Figure 2 d is a schematic diagram of the 3D hand reconstruction rendering. Figure 2 It can be seen that first, the bounding boxes of the hands and the object are determined according to the input frame image, i.e., the reference image. Then, the position between the hand bounding box and the object bounding box and other information can be used to further determine their layout representation. Since the hands are complex and changeable, more accurate 3D hand reconstruction renderings can also be used to display fine-grained structural information (such as hand motion posture and structural information). Figure 2 The arrow direction in the 3D hand reconstruction rendering in d is the direction of hand movement, where Figure 2 This is only an example and the present disclosure is not limited thereto.

[0062] It should be noted that Figure 2In the reference relationship graph, the bounding box size of the hand is fixed, while the bounding box of the original object can change adaptively with the position and posture of the object. Since the bounding box size of the hand is fixed, this feature has posture invariance and size invariance for the hand, but can only provide the position information of the hand. The size and posture information of the hand can be provided by the 3D hand reconstruction rendering, thereby realizing the decoupling of the hand and the object in the layout representation and improving the generation quality of the hand and the object.

[0063] It should be noted that Figure 2 The three-dimensional hand reconstruction rendering in the figure can be generated by a preset model or tool. For example, the hand can be reconstructed by a hand mesh (Hand Mesh), which is not limited in the present disclosure.

[0064] In the present disclosure, when the structural relationship processing network 102 fuses the reference relationship graph with the appearance features, it can adjust the bounding box of the object in the reference relationship graph to the bounding box of the target object based on the appearance features of the target object, thereby obtaining the interaction features between the hand and the target object.

[0065] The specific structure and type of the hand structure feature processing network 103 can be determined as needed. For example, the hand structure feature processing network 103 can be a control network ControlNet, which is not limited in the present disclosure.

[0066] Among them, the hand structure features may include features such as hand size and posture, which are not limited in the present disclosure.

[0067] Among them, the first feature map can be an interactive feature map for hand restoration.

[0068] In the present disclosure, the hand restoration network 104 can restore the hand at the position of the hand in the interaction feature based on the hand structure feature, the interaction feature and the generated hand texture feature, so as to obtain a first feature map of hand restoration.

[0069] The second feature map may be an interactive feature map including hand restoration and target object restoration.

[0070] In the present disclosure, the object restoration network 105 can restore the target object at the position of the target object in the first feature map based on the first feature map and the generated object texture features, thereby obtaining a second feature map in which both the hand and the target object are restored.

[0071] The specific type and structure of the decoder 106 can be determined as needed. For example, the decoder can be a VAE decoder, which is not limited in the present disclosure.

[0072] Among them, VAE is the abbreviation of Variational Autoencoder (VAE).

[0073] In some possible implementation forms, the interactive image generation model 100 provided by the present disclosure may also include a timing network connected to the object restoration network, wherein the timing network can be used to update the second feature map based on the output third feature map, thereby ensuring the continuity of the output video screen in the interactive video generation scenario and enhancing the anti-jitter capability of the interactive image generation model in timing.

[0074] The specific type and structure of the temporal network can be determined as needed. For example, the temporal network can be a temporal attention network (Temp Attention).

[0075] The third feature map may be a feature map before the second feature map.

[0076] It should be noted that in order to avoid the situation in which the temporal network fuses the features of the previous output image into the current image in the image generation scenario, the temporal network can determine whether to update the second feature map based on the features of the third feature map according to the time interval between the two acquired feature maps. The present disclosure does not limit this.

[0077] It should be noted that in the training process of the interactive image generation model, the temporal network and other models in the interactive image generation model are trained in batches. In the first stage of training, the training data of other networks are image data, while in the second stage of training, the training data of the temporal network is video data. Since video data has a temporal relationship, the temporal attention layer can be trained.

[0078] In the embodiment of the present disclosure, an interactive image generation model is proposed, including an object feature processing network, a structure relationship processing network, a hand structure feature processing network, a hand restoration network, an object restoration network and a decoder. First, an image containing a target object is processed by an object feature processing module to obtain the appearance features of the target object, and then the input reference relationship graph is fused with the appearance features by the structure relationship processing network to output the interactive features. Then, the hand structure features, interactive features and the generated hand texture features output by the hand structure feature processing network are fused by the hand restoration network to obtain a first feature map, and the first feature map and the generated object texture features are fused by the object restoration map to obtain a second feature map to be decoded, and finally the second feature map is decoded by the decoder to generate an interactive map containing the target object. Therefore, the interactive image generation model proposed in the present disclosure realizes the feature decoupling of the hand and the target object in the process of generating the interactive map, improves the generation quality of the hand and the object, and makes the generated interactive map effectively avoid the problems of object distortion and unreasonable interactive relationship, which provides conditions for improving the effect and quality of the generated interactive map.

[0079] It should be noted that the specific types and structures of the object feature processing network 101, the structural relationship processing network 102, the hand structure feature processing network 103, the hand restoration network 104, and the object restoration network 105 in the interactive image generation model provided by the present invention can be determined as needed.

[0080] Combine the following Figure 3 , the specific structure of each module in the interactive image generation model 300 proposed in the present disclosure is explained. Figure 3 A schematic diagram of the specific structure of each network in the interactive image generation model provided in an embodiment of the present disclosure.

[0081] like Figure 3 As shown, the interactive image generation model 300 includes: an object feature processing network 301, a structure relationship processing network 302, a hand structure feature processing network 303, a hand restoration network 304, an object restoration network 305 and a decoder 306.

[0082] In some possible implementations, the object feature processing network 301 may include a first encoder 3011 and a first convolutional network 3012 .

[0083] Among them, the first encoder 3011 can encode and extract the appearance features of the target object, and the first convolutional network 3012 can learn the appearance features of the target object.

[0084] The specific types and structures of the first encoder 3011 and the first convolutional network 3012 can be determined as needed. For example, the first encoder 3011 can be a VAE encoder, and the first convolutional network 3012 can be a Reference U-NET network, which is not limited in the present disclosure.

[0085] Among them, U-NET is a convolutional neural network based on deep learning, which can be used for image segmentation. It consists of two parts: an encoder and a decoder, and its shape is "U".

[0086] In some possible implementation forms, the structural relationship processing network 302 may include a second encoder 3021 and a second convolutional network 3022.

[0087] Among them, the second encoder 3021 can be used to encode and extract the interaction relationship between the hand and the original object in the reference relationship graph, and the second convolutional network 3022 can be used to denoise and fuse the reference relationship graph and appearance features.

[0088] The specific structure and type of the second encoder 3021 and the second convolutional network 3022 can be determined as needed. For example, the second encoder 3021 can be a layout encoder, and the second convolutional network 3022 can be a denoising convolutional network Denoising U-net, which is not limited in the present disclosure.

[0089] Among them, the output of the i-th convolutional layer of the first convolutional network 3012 is connected to the input of the i+1-th convolutional layer of the first convolutional network 3012 and the i+1-th convolutional layer of the second convolutional network 3022, respectively, and i is a positive integer.

[0090] It should be noted that the first convolutional network 3012 and the second convolutional network 3022 have the same number of convolutional layers and the same convolutional structure.

[0091] In the present disclosure, the appearance features of the target object are learned through the first encoder 3011 and the first convolutional network 3012, the interaction relationship in the reference relationship graph is learned through the second encoder 3021 and the second convolutional network 3022, and the reference relationship graph is merged with the appearance features of the target object to obtain the interaction features, thereby effectively avoiding the occurrence of unreasonable interaction relationships in the interaction features and improving the accuracy and reliability of the obtained interaction features.

[0092] In some possible implementation forms, the hand restoration network 304 may include a first storage body 3041 and a first attention network 3042. The i-th hand attention layer in the first attention network 3042 is used to fuse the i-th interaction feature output by the i-th convolution layer of the second convolution network 3022, the i-th hand structure feature output by the i-th network layer of the hand structure feature processing network 302, and the hand texture feature stored in the first storage body 3041 to obtain a first feature map of the i-th dimension.

[0093] Among them, the first storage bank 3041 can also be called "hand texture storage bank (Hand Texture MemoryBank)"; the first attention network 3042 can be a cross attention network (CrossAttention), which is not limited in the present disclosure.

[0094] In some possible implementation forms, the object restoration network 305 may include a second storage body 3051 and a second attention network 3052. The i-th object attention layer in the second attention network 3052 is used to fuse the output of the i-th hand attention layer of the second convolutional network 3022 and the object texture features stored in the second storage body 3051 to obtain a second feature map of the i-th dimension.

[0095] Among them, the second memory bank 3051 can also be called "object texture memory bank (Object TextureMemoryBank)"; the second attention network 3052 can be a cross attention network (CrossAttention), which is not limited in the present disclosure.

[0096] In the present disclosure, the hand restoration network 304 and the object restoration network 305 are used to restore the hand and the object texture respectively, thereby providing conditions for improving the restoration effect and accuracy.

[0097] It should be noted that the number of network layers included in the hand structure feature processing network 303 is the same as the number of attention layers in the hand restoration network 304 and the object restoration network 305.

[0098] It should be noted that the hand texture features stored in the first storage body 3041 and the object texture features stored in the second storage body 3051 can both be generated through training during the interactive image generation model training process.

[0099] In some possible implementation forms, the hand restoration network 304 may also include a first fusion layer connected to each hand attention layer, wherein the first fusion layer of the i-th dimension is used to fuse the first feature map of the i-th dimension output by the i-th hand attention layer, the i-th interaction feature and the hand mask map of the i-th dimension, thereby specifically enhancing the restoration of hand texture features.

[0100] It should be noted that the hand mask map can be obtained during the training of the interactive image generation model. During the model training process, the corresponding hand mask map can be obtained based on the hand bounding box. The hand mask map can also be scaled according to actual needs to obtain hand mask maps of different dimensions.

[0101] In some possible implementation forms, the object restoration network 305 may also include a second fusion layer connected to each object attention layer, wherein the second fusion layer of the i-th dimension is used to fuse the second feature map of the i-th dimension output by the i-th object attention layer, the first feature map of the i-th dimension, and the mask map corresponding to the target object, thereby improving the accuracy of texture restoration of the target object and enhancing the texture restoration of the target object.

[0102] It should be noted that the mask image corresponding to the target object can be obtained during the training process of the interactive image generation model.

[0103] In the disclosed embodiment, an image containing a target object is processed by a first encoder based on an object feature processing network and a first convolutional network to obtain appearance features of the target object, a reference relationship graph is fused with the appearance features by a second encoder of a structural relationship combing network and a second convolutional network to obtain interaction features, hand structure features, interaction features and hand texture features are fused by a first storage body of a hand restoration network and a first attention network to obtain a first feature map, the first feature map and object texture features are fused by a second storage body of the object restoration network and a second attention network to obtain a second feature map, and finally the second feature map is decoded by a decoder to generate an interaction map containing the target object, thereby improving the accuracy and generation effect of the interactive image generation model.

[0104] Combine the following Figure 4 , the training process of the interactive image generation model provided in the embodiment of the present disclosure is illustrated by way of example, Figure 4 A schematic diagram of the training process of the interactive image generation model provided in an embodiment of the present disclosure.

[0105] Figure 4In the figure, an object feature processing network includes a first encoder 401 and a first convolutional network 402, the first convolutional network 402 is a Reference U-NET, a structural relationship processing network includes a second encoder 404 and a second convolutional network 405, the second convolutional network 405 is a Denoising U-net, a hand restoration network 407 includes a first storage body 409, a first attention network 411 and a first fusion layer 410, and an object restoration network 408 includes a second storage body 412, a second attention network 414 and a second fusion layer 413.

[0106] Among them, Self Attention is the self-attention layer, Temp Attention is the temporal attention layer, HandAttention is the hand attention layer, and Obj Attention is the object attention layer.

[0107] It should be noted that since the first attention network 411 in the hand restoration network 407 and the second attention network 414 in the object restoration network 408 can be seamlessly inserted into the second convolutional network 405 for use, the Hand Attention in the second convolutional network 405 is the first attention network 411 in the hand restoration network 407, and the Obj Attention is the second attention network 414 in the object restoration network 408.

[0108] like Figure 4 As shown, when training the interactive image generation model, it can be divided into two parts: hand and object. The appearance features of the target object are extracted by the first encoder 401, and the appearance features of the target object are learned by the first convolutional network 402. The interactive information in the reference relationship graph is extracted by the second encoder 404, and the three-dimensional rendering of the hand is obtained by the hand structure feature processing network 403, and the interactive information (such as motion and structure) in the reference relationship graph is learned by the second convolutional network 405.

[0109] It should be noted that the first convolutional network 402 and the second convolutional network 405 use the same U-Net convolutional structure, and a cross-attention mechanism can be used for information interaction on the output feature maps of corresponding dimensions.

[0110] Furthermore, in the interactive image generation model training process provided by the present disclosure, the hand restoration network 407 and the object restoration network 408 may also be trained. Figure 4In the present invention, when the hand restoration network 407 and the object restoration network 408 are trained, in the first attention network 411, the i-th hand structure feature output by the i-th network layer of the hand structure feature processing network 403 can be used as a query, the hand texture feature stored in the first storage body 409 can be used as a key and a value, and the i-th interaction feature output by the i-th convolution layer of the second convolution network 405 can be fused to obtain a first feature map of the i-th dimension. In the second attention network 414, the output of the i-th hand attention layer of the second convolution network 405 can be used as a query, and the object texture feature stored in the second storage body 412 can be used as a key and a value to obtain a second feature map of the i-th dimension.

[0111] Furthermore, when training the hand restoration network 407 and the object restoration network 408, in order to enhance the restoration of hand texture and object texture, the first feature map of the i-th dimension, the i-th interaction feature and the hand mask map of the i-th dimension can be fused, so that the first attention network 411 can well fuse the hand structure features and the hand texture, and the second feature map of the i-th dimension, the first feature map of the i-th dimension and the mask map corresponding to the target object are fused, so that the second attention network 414 can perform targeted texture restoration in the object area, and the present disclosure does not limit this.

[0112] Furthermore, in order to improve the continuity and stability of the generated video images, a learnable temporal network, i.e., a temporal attention layer, may be added to the second convolutional network 405 to train the anti-jitter capability of the interactive image generation model in terms of timing.

[0113] It should be noted that when training the temporal network, the temporal network is trained in batches with other networks. The training data of the temporal network is video data, and the training data of other networks is image data.

[0114] In the present disclosure, in order to avoid unreasonable contact or size on the physical level due to the large difference between the target object and the original object in the generated interaction image, the interaction image generation model may also include a preprocessing component for adjusting the first bounding box corresponding to the original object in the reference relationship diagram and / or the second bounding box corresponding to the holding hand based on the size ratio between the target object and the original object.

[0115] It should be noted that the first bounding box corresponding to the original object and the second bounding box corresponding to the hand holding the object can both be maximum circumscribed rectangular boxes, and the present disclosure does not limit this.

[0116] like Figure 5 As shown, Figure 5A schematic diagram of a flow chart of processing a reference relationship graph by a preprocessing component in an interactive image generation model provided in an embodiment of the present disclosure.

[0117] Step 501, analyzing the reference relationship diagram to determine a first distance between the holding hand and the original object.

[0118] The first distance may be the distance between an actual contact point on the edge of the second bounding box of the object-holding hand and an actual contact point on the first bounding box of the original object.

[0119] The contact point may be a center point on the edge of each preset bounding box.

[0120] In the present disclosure, when the preprocessing component adjusts the first bounding box corresponding to the original object in the reference relationship graph, and / or the second bounding box corresponding to the holding hand, it can first parse the reference relationship graph, determine the actual contact points of the first bounding box and the second bounding box from the center points on each edge of the first bounding box and the center points on each edge of the second bounding box, and then determine the distance between the actual contact points on the first bounding box and the second bounding box, and determine it as the first distance between the holding hand and the original object.

[0121] It should be noted that when determining the first distance between the holding hand and the original object, when there are multiple actual contact points between the first bounding box and the second bounding box, multiple first distances can be determined based on the actual contact points on the first bounding box and the actual contact points on the second bounding box. The present disclosure does not limit this.

[0122] Step 502 : Based on the size ratio between the target object and the original object, the first bounding box corresponding to the original object is resized to obtain a third bounding box.

[0123] The third bounding box is the bounding box of the target object.

[0124] In the present disclosure, in order to avoid unreasonable interaction caused by a large size difference between the target object and the original object, the preprocessing component can first determine the size ratio between the target object and the original object, and then adjust the size of the first bounding box based on the size ratio to obtain an adjusted third bounding box.

[0125] Step 503: Based on the distance, adjust the position of the second bounding box corresponding to the object holding hand, wherein the second distance between the adjusted second bounding box and the third bounding box is equal to the first distance.

[0126] In the present disclosure, after the first bounding box is resized to obtain the third bounding box, the relative position between the third bounding box and the second bounding box may change. At this time, the position of the second bounding box can be adjusted based on the first distance so that the second distance between the adjusted second bounding box and the third bounding box is equal to the first distance.

[0127] It should be noted that, in the case where there are multiple first distances, each second distance between the adjusted second bounding box and the third bounding box needs to be the same as the corresponding first distance.

[0128] In some possible implementations, when the first bounding box is resized, in order to avoid the situation where the adjusted object stops in mid-air, etc., the preprocessing component may also move the center point on the designated side of the third bounding box to coincide with the center point on the designated side of the first bounding box. For example, the center point on the bottom side of the third bounding box may be moved to coincide with the center point on the bottom side of the first bounding box, etc., which is not limited in the present disclosure.

[0129] In the disclosed embodiment, through the preprocessing component in the interactive image generation model, the first bounding box corresponding to the original object in the reference relationship diagram and / or the second bounding box corresponding to the holding hand are adjusted based on the size ratio between the target object and the original object, thereby effectively avoiding the problem of unreasonable interaction caused by the large size difference between the target object and the original object, and providing conditions for improving the interactive image generation effect and quality of the interactive image generation model.

[0130] Combine the following Figure 6 , an example is given to illustrate the process of processing the reference relationship graph by the preprocessing component in the interactive image generation model disclosed in the present invention. Figure 6 A schematic diagram of the process of processing a reference relationship graph by a preprocessing component in an interactive image generation model provided in an embodiment of the present disclosure.

[0131] Figure 6 middle, Figure 6 A to Figure 6 B is a group of hands holding Figure 6 The object in A becomes two-handed Figure 6 The schematic diagram of the target object in B, the schematic diagrams of label 601 to label 604 are Figure 6 A becomes Figure 6 The adjustment diagram of the reference relationship diagram corresponding to B, Figure 6 C to Figure 6 D is a group of single-handed Figure 6 The object in C becomes one-handed Figure 6 The schematic diagram of the target object in D, the schematic diagrams of label 605 to label 608 are Figure 6 C becomes Figure 6D corresponding to the reference relationship diagram adjustment schematic diagram, wherein, in the schematic diagrams numbered 601 to 608, the boundary box 6a is Figure 6 The bounding box of the object in A, the bounding box 6b and the bounding box 6c are both the hand bounding boxes, and the bounding box 6d is Figure 6 The bounding box of the target object in B, the bounding box 6e is Figure 6 The bounding box of the object in C, the bounding box 6f is Figure 6 The bounding box of the target object in D.

[0132] like Figure 6 As shown, the preprocessing component in the interactive image generation model of the present disclosure first determines the actual contact points from the center points on the edges of the hand bounding box and the center points on the edges of the object bounding box, such as Figure 6 The points with light-colored circles shown in the schematic diagrams of label 601 and label 605 are the actual contact points. Then, based on the size ratio between the target object and the original object, the bounding box of the original object is adjusted to the bounding box of the target object, such as adjusting the size and shape of the bounding box 6a to the size and shape of the bounding box 6d as shown in the schematic diagram of label 602, and adjusting the size and shape of the bounding box 6e to the size and shape of the bounding box 6f as shown in the schematic diagram of label 606. Then, the distance between the hand bounding box and the target object bounding box is adjusted so that the adjusted distance is equal to the distance between the contact points of the original object and the hand, such as adjusting the positions of the bounding boxes 6c and 6b as shown in the schematic diagram of label 603, and adjusting the positions of the bounding boxes 607 and 610 as shown in the schematic diagram of label 608. In the schematic diagram, the position of the bounding box 6c is adjusted so that the distance between the contact point on the bounding box of the target object and the contact point on the hand bounding box is adjusted to the distance between the original object and the hand holding the object. Finally, the center point on the bottom edge of the bounding box of the target object is moved to coincide with the center point on the bottom edge of the bounding box of the original object, such as the bottom center point of the bounding box 6d in the schematic diagram labeled 604 coincides with the bottom center point of the bounding box 6a, and the bottom center point of the bounding box 6f in the schematic diagram labeled 608 coincides with the bottom center point of the bounding box 6e. This can effectively avoid the problem of unreasonable contact due to the large size difference between the target object and the original object, thereby providing conditions for improving the reliability of the interactive image generation model in generating interactive images.

[0133] Figure 7 A schematic flow chart of a method for generating an interactive image provided by an embodiment of the present disclosure.

[0134] like Figure 7 As shown, the method for generating the interactive image may include the following steps:

[0135] Step 701, obtaining a reference relationship graph, a target object graph and hand structure features.

[0136] In the present disclosure, before generating an interactive image, in order to improve the reliability and accuracy of the interactive image generation, a reference relationship graph, a target object graph and a hand structure feature graph may be first obtained.

[0137] Step 702: Process the target object image to determine the appearance features of the target object.

[0138] In the present disclosure, after obtaining the reference relationship graph, the target object graph and the hand structure features, the target object graph can be processed by the object feature processing network in the interactive image generation model proposed in the above embodiments of the present disclosure to determine the appearance features of the target object.

[0139] In the present disclosure, when the target object image is processed by the object feature processing network, the first encoder and the first convolutional network in the object feature processing network can be used in sequence to encode and convolve the target object image to obtain the appearance features of the target object.

[0140] Step 703: Process the reference relationship graph and the appearance features to determine the interaction features.

[0141] In the present disclosure, after the appearance features of the target object are determined, the reference relationship graph and the appearance features can be processed through the structural relationship processing network in the interactive image generation model to determine the interactive features.

[0142] In the present disclosure, when the reference relationship graph and the appearance features are processed by the structural relationship processing network, the reference relationship graph can be first encoded by the second encoder in the structural relationship processing network to obtain the initial interaction features, and then the i+1th interaction feature output by the i-th convolutional layer in the second convolutional network in the structural relationship processing network and the appearance feature output by the i-th convolutional layer in the first convolutional network are convolved to obtain the i+1th interaction feature, where i is an integer, thereby improving the reliability of the obtained interaction features.

[0143] Step 704: Fusing the interaction features, the preset hand texture features, and the hand structure features to obtain a first feature map.

[0144] It should be noted that the preset hand texture features can be pre-stored in the first storage volume in the hand restoration network.

[0145] In the present disclosure, after determining the interaction features, the interaction features, preset hand texture features and hand structure features can be fused through the hand restoration network in the interactive image generation model to obtain a first feature map of hand restoration.

[0146] In the present disclosure, when the interaction features, preset hand texture features and hand structure features are fused through the hand restoration network, the i-th hand attention layer in the first attention network in the hand restoration network can be used to fuse the i-th interaction features output by the i-th convolutional layer of the second convolutional network, the i-th hand structure features and hand texture features output by the i-th network layer in the hand structure feature processing network to obtain the first feature map of the i-th dimension.

[0147] In some possible implementation forms, in order to perform enhanced restoration on the hands in a targeted manner, after obtaining the first feature map of the i-th dimension, the first fusion layer of the i-th dimension in the first fusion network can be used to fuse the first feature map of the i-th dimension, the i-th interaction feature and the hand mask map of the i-th dimension to obtain an updated first feature map of the i-th dimension.

[0148] Step 705: Fusing the first feature map and the preset object texture feature map to obtain a second feature map.

[0149] It should be noted that the preset object texture features can be pre-stored in the second storage volume in the object restoration network.

[0150] In the present disclosure, after obtaining the first feature map, the first feature map and the preset object texture feature map can be fused through the object restoration network in the interactive image generation model to obtain a second feature map for hand restoration and object restoration.

[0151] In the present disclosure, when the first feature map and the preset object texture feature map are fused through the object restoration network, the i-th object attention layer in the second attention network in the object restoration network can be used to fuse the output of the i-th hand attention layer of the second convolutional network and the object texture feature to obtain the second feature map of the i-th dimension.

[0152] In some possible implementation forms, in order to enhance the effect and accuracy of texture restoration of the target object, the second fusion layer of the i-th dimension in the second fusion network can also be used to fuse the second feature map of the i-th dimension, the updated first feature map of the i-th dimension, and the mask map corresponding to the target object to obtain the updated second feature map of the i-th dimension.

[0153] In the present disclosure, after obtaining the second feature map of the i-th dimension, the second feature map of the i-th dimension and the appearance features of the i-th dimension can be used as inputs of the structural relationship processing network of the i+1-th dimension to obtain the interactive features of the i+1-th dimension. The present disclosure does not limit this.

[0154] Step 706: Generate an interaction graph including the target object based on the second feature graph.

[0155] In the present disclosure, after obtaining the second feature map, the second feature map can be decoded by a decoder in the interactive image generation model to generate an interactive map containing the target object.

[0156] In the embodiment of the present disclosure, firstly, a reference relationship graph, a target object graph and a hand structure feature graph are obtained, and the target object graph is processed to determine the appearance features of the target object, and then the reference relationship graph and the appearance features are processed to determine the interaction features, and the interaction features, the preset hand texture features and the hand structure features are fused to obtain a first feature graph, and then the first feature graph and the preset object texture feature graph are fused to obtain a second feature graph, and finally based on the second feature graph, an interaction graph containing the target object is generated. Thus, by respectively inputting the reference relationship graph, the target object graph and the hand structure feature graph into the interactive image generation model proposed in the above embodiment of the present disclosure, and processing them using the corresponding network structure in the interactive image generation model, an interaction graph containing the target object is generated, thereby effectively avoiding the generation of unreasonable character interaction relationships in the generated interactive graph, and improving the generation quality and accuracy of the interactive image.

[0157] Figure 8 A schematic flow chart of a method for generating an interactive image provided by an embodiment of the present disclosure.

[0158] like Figure 8 As shown, the method for generating the interactive image may include the following steps:

[0159] Step 801, obtaining a reference relationship graph, a target object graph and hand structure features.

[0160] Step 802: Process the target object image to determine the appearance features of the target object.

[0161] Step 803: Process the reference relationship graph and the appearance features to determine the interaction features.

[0162] Step 804: Fusing the interaction features, the preset hand texture features, and the hand structure features to obtain a first feature map.

[0163] Step 805: Fusing the first feature map and the preset object texture feature map to obtain a second feature map.

[0164] The specific implementation forms of steps 801 to 805 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0165] Step 806, when the reference relationship graph is an image frame in a reference video, the second feature graph is updated based on a third feature graph associated with other image frames in the output reference video to obtain an updated second feature graph.

[0166] Among them, the reference video can be determined according to actual needs.

[0167] In the present disclosure, after obtaining the second feature map, when the reference relationship map is an image frame in a reference video frame, in order to improve the accuracy and reliability of the second feature map, the second feature map can be updated based on a third feature map associated with other image frames in the output reference video to obtain an updated second feature map.

[0168] Step 807: Generate an interaction graph including the target object based on the updated second feature graph.

[0169] In the present disclosure, after the second feature map is updated to obtain the updated second feature map, an interaction map including the target object can be generated based on the updated second feature map, thereby improving the accuracy and reliability of the generated interaction map.

[0170] In the disclosed embodiment, a reference relationship graph, a target object graph and a hand structure feature are first obtained, and then the target object graph is processed to determine the appearance feature of the target object, and the reference relationship graph and the appearance feature are processed to determine the interaction feature, and then the interaction feature, the preset hand texture feature and the hand structure feature are fused to obtain a first feature graph, and the first feature graph and the preset object texture feature graph are fused to obtain a second feature graph, and finally, when the reference relationship graph is an image frame in the reference video, the second feature graph is updated based on the third feature graph associated with other image frames in the output reference video to obtain an updated second feature graph, and based on the updated second feature graph, an interaction graph containing the target object is generated. Thus, after the reference relationship graph, the target object graph and the hand structure feature are respectively input into the interaction image generation model to obtain the second feature graph containing hand restoration and object restoration, when the reference relationship graph is an image frame in the reference video, the second feature graph is updated based on the third feature graph of other image frames in the output reference video, and the interaction graph is actively generated with the updated second feature graph, thereby improving the accuracy and reliability of the interaction image generation.

[0171] Fig. 9 A schematic flow chart of a method for generating an interactive image provided by an embodiment of the present disclosure.

[0172] like Fig. 9 As shown, the method for generating the interactive image may include the following steps:

[0173] Step 901, obtaining a reference relationship graph, a target object graph and hand structure features.

[0174] Step 902: Process the target object image to determine the appearance features of the target object.

[0175] The specific implementation forms of step 901 to step 902 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0176] Step 903: Based on the size ratio between the target object and the original object in the reference relationship graph, adjust the first bounding box corresponding to the original object in the reference relationship graph and / or the second bounding box corresponding to the holding hand.

[0177] In the present disclosure, after determining the appearance features of the target object, and before determining the interaction features based on the reference relationship graph and the appearance features, in order to avoid the problem of unreasonable interaction relationships in the interaction features due to the size deviation between the target object and the original object in the reference relationship graph, the preprocessing component in the interaction image generation model can first be used to adjust the first bounding box corresponding to the original object in the reference relationship graph and / or the second bounding box corresponding to the holding hand based on the size ratio between the target object and the original object in the reference relationship graph.

[0178] Step 904: Process the reference relationship graph and the appearance features to determine the interaction features.

[0179] In the present disclosure, after adjusting the first bounding box and / or the second bounding box in the reference relationship graph, the reference relationship graph and the appearance features can be processed through a structural relationship processing network to determine the interaction features, thereby improving the reliability of the determined interaction features.

[0180] Step 905: Fusing the interaction features, the preset hand texture features, and the hand structure features to obtain a first feature map.

[0181] Step 906: Fusing the first feature map and the preset object texture feature map to obtain a second feature map.

[0182] Step 907: Generate an interaction graph including the target object based on the second feature graph.

[0183] Among them, the specific implementation forms of steps 904 to 907 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be repeated here.

[0184] In the disclosed embodiment, a reference relationship graph, a target object graph and hand structure features are first obtained, and the target object graph is processed to determine the appearance features of the target object. Then, based on the size ratio between the target object and the original object in the reference relationship graph, the first bounding box corresponding to the original object in the reference relationship graph and / or the second bounding box corresponding to the holding hand are adjusted. Thereafter, the reference relationship graph and the appearance features are processed to determine the interaction features, and the interaction features, preset hand texture features and hand structure features are fused to obtain a first feature graph. Finally, the first feature graph and the preset object texture feature graph are fused to obtain a second feature graph, and based on the second feature graph, an interaction graph including the target object is generated. Therefore, after determining the appearance features of the target object based on the target object map, the object boundary box and / or the holding hand boundary box in the reference relationship map are adjusted based on the size ratio between the target object and the original object in the reference relationship map, and the interaction features are determined based on the adjusted reference relationship map and the appearance features. The interaction features, hand texture features and hand structure features are fused to obtain a first feature map, the first feature map is fused with the object texture feature map to obtain a second feature map, and an interaction map is generated based on the second feature map, thereby effectively avoiding unreasonable interaction relationships in the generated interaction map and improving the reliability of interactive image generation.

[0185] Fig.10 A schematic flow chart of a method for generating an interactive image provided by an embodiment of the present disclosure.

[0186] like Fig.10 As shown, the method for generating the interactive image may include the following steps:

[0187] Step 1001, obtaining a reference relationship graph, a target object graph and hand structure features.

[0188] Step 1002: Process the target object image to determine the appearance features of the target object.

[0189] The specific implementation forms of step 1001 to step 1002 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0190] Step 1003: parse the reference relationship graph to determine a first distance between the holding hand and the original object.

[0191] In the present disclosure, after determining the appearance features of the target object, in order to improve the reliability of adjusting the reference relationship graph, the reference relationship graph can be first analyzed by a preprocessing component to determine the first distance between the holding hand and the original object.

[0192] It should be noted that when there are multiple contact points on the edge of the second bounding box of the holding hand and the edge of the first bounding box of the original object, multiple first distances between the contact points on the edge of the second bounding box and the contact points on the first bounding box can be determined, and the present disclosure does not limit this.

[0193] Step 1004 , based on the size ratio between the target object and the original object, the first bounding box is resized to obtain a third bounding box.

[0194] In the present disclosure, after determining the first distance between the holding hand and the original object, in order to avoid unreasonable contact or interaction due to the size difference between the target object and the original object, the size ratio between the target object and the original object can be first determined, and then based on the size ratio, the first bounding box can be resized to obtain a third bounding box.

[0195] Step 1005: adjusting the position of the second bounding box based on the distance, wherein the second distance between the adjusted second bounding box and the third bounding box is equal to the first distance.

[0196] In the present disclosure, after the first bounding box is resized to obtain the third bounding box, the position of the second bounding box can be adjusted based on the first distance between the holding hand and the original object, so that the second distance between the adjusted second bounding box and the third bounding box is equal to the first distance.

[0197] It should be noted that, when there are multiple first distances, when adjusting the position of the second bounding box, each second distance between the adjusted second bounding box and the third bounding box needs to be equal to the corresponding first distance, and the present disclosure does not limit this.

[0198] In some possible implementations, after adjusting the position of the second bounding box based on the distance, in order to prevent the third bounding box from being in an unreasonable position, such as staying in mid-air, etc., the center point of the designated side of the third bounding box may be moved to coincide with the center point of the designated side of the first bounding box. For example, the center point of the bottom side of the third bounding box may be moved to coincide with the center point of the bottom side of the first bounding box, which is not limited in the present disclosure.

[0199] Step 1006: Process the reference relationship graph and the appearance features to determine the interaction features.

[0200] Step 1007, the interaction feature, the preset hand texture feature and the hand structure feature are integrated to obtain a first feature map.

[0201] Step 1008: Fusing the first feature map and the preset object texture feature map to obtain a second feature map.

[0202] Step 1009: Generate an interaction graph including the target object based on the second feature graph.

[0203] Among them, the specific implementation forms of steps 1006 to 1009 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be repeated here.

[0204] In the disclosed embodiment, a reference relationship graph, a target object graph and hand structure features are first obtained, and the target object graph is processed to determine the appearance features of the target object. Then, the reference relationship graph is parsed to determine a first distance between the holding hand and the original object. Based on the size ratio between the target object and the original object, the size of the first bounding box is adjusted to obtain a third bounding box. Then, based on the distance, the position of the second bounding box is adjusted. The reference relationship graph and the appearance features are processed to determine the interaction features. Finally, the interaction features, preset hand texture features and hand structure features are fused to obtain a first feature graph. The first feature graph and the preset object texture feature graph are fused to obtain a second feature graph. Based on the second feature graph, an interaction graph including the target object is generated. Therefore, by determining the appearance features of the target object based on the target object map, parsing the reference relationship map and determining the distance between the holding hand and the original object, adjusting the object boundary box in the reference relationship map based on the size ratio between the target object and the original object, and adjusting the position of the holding hand boundary box based on the distance, determining the interaction features based on the adjusted reference relationship map and appearance features, and generating an interaction feature map based on the interaction features, hand texture features, hand structure features and object texture features, and generating an interaction map based on the interaction feature map, thereby improving the reliability of the generated interaction image and improving the accuracy of the interaction image generation.

[0205] In order to implement the above embodiment, the present disclosure also proposes a device for generating an interactive image.

[0206] Fig.11 A schematic diagram of the structure of an interactive image generation device provided in an embodiment of the present disclosure.

[0207] like Fig.11 As shown, the interactive image generating device 1100 includes: a first acquisition module 1101 , a first determination module 1102 , a second determination module 1103 , a second acquisition module 1104 , a third acquisition module 1105 , and a generation module 1106 .

[0208] A first acquisition module 1101 is used to acquire a reference relationship graph, a target object graph and hand structure features;

[0209] A first determination module 1102 is used to process the target object image to determine the appearance features of the target object;

[0210] The second determination module 1103 is used to process the reference relationship graph and the appearance features to determine the interaction features;

[0211] The second acquisition module 1104 is used to fuse the interaction feature, the preset hand texture feature and the hand structure feature to obtain a first feature map;

[0212] The third acquisition module 1105 is used to fuse the first feature map and the preset object texture feature map to obtain a second feature map;

[0213] The generating module 1106 is used to generate an interaction graph including the target object based on the second feature graph.

[0214] In a possible implementation of the present disclosure, the generating module 1106 is specifically configured to:

[0215] When the reference relationship graph is an image frame in a reference video, the second feature graph is updated based on a third feature graph associated with other image frames in the output reference video to obtain an updated second feature graph;

[0216] Based on the updated second feature map, an interaction map including the target object is generated.

[0217] In a possible implementation of the present disclosure, the first determining module 1102 is specifically configured to:

[0218] Using the first encoder and the first convolutional network in sequence, encode and convolve the target object image to obtain the appearance features of the target object;

[0219] The second determination module is specifically used for:

[0220] Encoding the reference relationship graph using a second encoder to obtain initial interaction features;

[0221] Using the i+1th convolutional layer in the second convolutional network, the interaction features output by the i+1th convolutional layer in the first convolutional network and the appearance features output by the i-th convolutional layer in the first convolutional network are convolved to obtain the i-th interaction feature, where i is a positive integer.

[0222] In a possible implementation of the present disclosure, the second acquisition module 1104 is specifically configured to:

[0223] Using the i-th hand attention layer in the first attention network, the i-th interaction feature output by the i-th convolutional layer of the second convolutional network, the i-th hand structure feature output by the i-th network layer in the hand structure feature processing network and the hand texture feature are fused to obtain the first feature map of the i-th dimension;

[0224] The third acquisition module 1105 is specifically used for:

[0225] Using the i-th object attention layer in the second attention network, the output of the i-th hand attention layer of the second convolutional network and the object texture features are fused to obtain the second feature map of the i-th dimension.

[0226] In a possible implementation of the present disclosure, the second acquisition module 1104 is further configured to:

[0227] Using the first fusion layer of the i-th dimension in the first fusion network, the first feature map of the i-th dimension, the i-th interaction feature and the hand mask map of the i-th dimension are fused to obtain an updated first feature map of the i-th dimension.

[0228] In a possible implementation of the present disclosure, the second acquisition module 1104 is further configured to:

[0229] Using the second fusion layer of the i-th dimension in the second fusion network, the second feature map of the i-th dimension, the updated first feature map of the i-th dimension, and the mask map corresponding to the target object are fused to obtain the updated second feature map of the i-th dimension.

[0230] In a possible implementation of the present disclosure, the second determining module 1103 is further configured to:

[0231] Based on the size ratio between the target object and the original object in the reference relationship graph, the first bounding box corresponding to the original object in the reference relationship graph and / or the second bounding box corresponding to the holding hand are adjusted.

[0232] In a possible implementation of the present disclosure, the second determining module 1103 is further configured to:

[0233] Parsing the reference relationship graph to determine a first distance between the holding hand and the original object;

[0234] Based on the size ratio between the target object and the original object, the first bounding box is resized to obtain a third bounding box;

[0235] Based on the distance, the position of the second bounding box is adjusted, wherein a second distance between the adjusted second bounding box and the third bounding box is equal to the first distance.

[0236] In a possible implementation of the present disclosure, the above-mentioned device further includes:

[0237] The moving module (not shown in the figure) is used to move the center point on the designated side of the third bounding box to coincide with the center point on the designated side of the first bounding box.

[0238] The functions and specific implementation principles of the above modules in the embodiments of the present disclosure can be referred to the above method embodiments, and will not be repeated here.

[0239] In the embodiment of the present disclosure, firstly, a reference relationship graph, a target object graph and a hand structure feature graph are obtained, and the target object graph is processed to determine the appearance features of the target object, and then the reference relationship graph and the appearance features are processed to determine the interaction features, and the interaction features, the preset hand texture features and the hand structure features are fused to obtain a first feature graph, and then the first feature graph and the preset object texture feature graph are fused to obtain a second feature graph, and finally based on the second feature graph, an interaction graph containing the target object is generated. Thus, by respectively inputting the reference relationship graph, the target object graph and the hand structure feature graph into the interactive image generation model proposed in the above embodiment of the present disclosure, and processing them using the corresponding network structure in the interactive image generation model, an interaction graph containing the target object is generated, thereby effectively avoiding the generation of unreasonable character interaction relationships in the generated interactive graph, and improving the generation quality and accuracy of the interactive image.

[0240] Fig.12 A schematic block diagram of an example electronic device 1200 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0241] like Fig.12 As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the device 1200 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0242] A number of components in the device 1200 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0243] The computing unit 1201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above, such as the generation method of the interactive image. For example, in some embodiments, the generation method of the interactive image may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the generation method of the interactive image described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to execute the method for generating an interactive image in any other appropriate manner (for example, by means of firmware).

[0244] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0245] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a general-purpose computer, a special-purpose computer, or other programmable processor or controller, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0246] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0247] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0248] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0249] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0250] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0251] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the description of the present disclosure, the words "if" and "if" used can be interpreted as "at the time of" or "when" or "in response to determination" or "under the circumstances of".

[0252] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An interactive image generation model, comprising: Object feature processing network, structure relationship processing network, hand structure feature processing network, hand restoration network, object restoration network and decoder; The object feature processing network is used to process an image containing a target object to obtain appearance features of the target object; The structural relationship processing network is used to fuse the input reference relationship graph with the appearance feature and output an interactive feature; The hand restoration network is used to fuse the hand structure features output by the hand structure feature processing network, the interaction features and the generated hand texture features to obtain a first feature map; The object restoration network is used to fuse the first feature map and the generated object texture features to obtain a second feature map to be decoded; The decoder is used to decode the second feature map to generate an interaction map including the target object.

2. The model of claim 1, wherein: The model also includes: a temporal network connected to the object restoration network; The timing network is used to update the second feature map based on the output third feature map.

3. The model of claim 1, wherein: The object feature processing network includes a first encoder and a first convolutional network; the structural relationship processing network includes a second encoder and a second convolutional network, wherein the output of the i-th convolutional layer of the first convolutional network is respectively connected to the input of the i+1-th convolutional layer of the first convolutional network and the i+1-th convolutional layer of the second convolutional network, and i is a positive integer.

4. The model of claim 3, wherein: The hand restoration network includes a first storage body and a first attention network, wherein the i-th hand attention layer in the first attention network is used to fuse the i-th interaction feature output by the i-th convolutional layer of the second convolutional network, the i-th hand structure feature output by the i-th network layer of the hand structure feature processing network, and the hand texture feature stored in the first storage body to obtain a first feature map of the i-th dimension; The object restoration network includes a second storage body and a second attention network. The i-th object attention layer in the second attention network is used to fuse the output of the i-th hand attention layer of the second convolutional network and the object texture features stored in the second storage body to obtain a second feature map of the i-th dimension.

5. The model of claim 4, wherein: The hand restoration network also includes a first fusion layer connected to each hand attention layer; The first fusion layer of the i-th dimension is used to fuse the first feature map of the i-th dimension output by the i-th hand attention layer, the i-th interaction feature and the hand mask map of the i-th dimension.

6. The model of claim 4, wherein: The object restoration network also includes a second fusion layer connected to each object attention layer; The second fusion layer of the i-th dimension is used to fuse the second feature map of the i-th dimension output by the i-th object attention layer, the first feature map of the i-th dimension and the mask map corresponding to the target object.

7. A model as claimed in any one of claims 1 to 6, wherein: The model also includes: a preprocessing component; The preprocessing component is used to adjust the first bounding box corresponding to the original object in the reference relationship diagram and / or the second bounding box corresponding to the holding hand based on the size ratio between the target object and the original object.

8. The model of claim 7, wherein: The pre-processing component is specifically used for: Parsing the reference relationship graph to determine a first distance between the holding hand and the original object; Based on a size ratio between the target object and the original object, resizing the first bounding box to obtain a third bounding box; Based on the distance, the position of the second bounding box is adjusted, wherein a second distance between the adjusted second bounding box and the third bounding box is equal to the first distance.

9. The model of claim 8, wherein: The preprocessing component is further used to: move the center point on the designated side of the third bounding box to coincide with the center point on the designated side of the first bounding box.

10. A method for generating an interactive image, comprising: Obtain reference relationship graph, target object graph and hand structure features; Processing the target object image to determine the appearance features of the target object; Processing the reference relationship graph and the appearance features to determine interaction features; The interaction feature, the preset hand texture feature and the hand structure feature are integrated to obtain a first feature map; The first feature map and the preset object texture feature map are merged to obtain a second feature map; Based on the second feature map, an interaction map including the target object is generated.

11. The method of claim 10, wherein: The step of generating an interaction graph including the target object based on the second feature graph includes: In a case where the reference relationship graph is an image frame in a reference video, the second feature graph is updated based on a third feature graph associated with other image frames in the reference video that has been output, to obtain an updated second feature graph; Based on the updated second feature map, an interaction map including the target object is generated.

12. The method of claim 10, wherein: The processing of the target object image to determine the appearance features of the target object includes: Using a first encoder and a first convolutional network in sequence, encoding and convolution processing are performed on the target object image to obtain appearance features of the target object; The processing of the reference relationship graph and the appearance features to determine the interaction features includes: Encoding the reference relationship graph using a second encoder to obtain the initial interaction feature; Using the i+1th convolutional layer in the second convolutional network, convolve the i-th interaction feature output by the i-th convolutional layer in the second convolutional network and the appearance feature output by the i-th convolutional layer in the first convolutional network to obtain the i+1th interaction feature, where i is a positive integer.

13. The method of claim 12, wherein: The step of fusing the interaction feature, the preset hand texture feature and the hand structure feature to obtain a first feature map includes: Using the i-th hand attention layer in the first attention network, the i-th interaction feature output by the i-th convolutional layer of the second convolutional network, the i-th hand structure feature output by the i-th network layer in the hand structure feature processing network, and the hand texture feature are fused to obtain a first feature map of the i-th dimension; The step of fusing the first feature map with a preset object texture feature map to obtain a second feature map includes: Using the i-th object attention layer in the second attention network, the output of the i-th hand attention layer of the second convolutional network and the object texture features are fused to obtain a second feature map of the i-th dimension.

14. The method of claim 13, wherein: After obtaining the first feature map of the i-th dimension, the method further includes: Using the first fusion layer of the i-th dimension in the first fusion network, the first feature map of the i-th dimension, the i-th interaction feature and the hand mask map of the i-th dimension are fused to obtain an updated first feature map of the i-th dimension.

15. The method of claim 14, wherein: After obtaining the second feature map of the i-th dimension, the method further includes: Using the second fusion layer of the i-th dimension in the second fusion network, the second feature map of the i-th dimension, the updated first feature map of the i-th dimension, and the mask map corresponding to the target object are fused to obtain an updated second feature map of the i-th dimension.

16. The method according to any one of claims 10 to 15, wherein: Before the reference relationship graph and the appearance features are processed to determine the interaction features, the method further includes: Based on the size ratio between the target object and the original object in the reference relationship diagram, the first bounding box corresponding to the original object in the reference relationship diagram and / or the second bounding box corresponding to the holding hand are adjusted.

17. The method of claim 16, wherein: The adjusting of the first bounding box corresponding to the original object in the reference relationship graph and / or the second bounding box corresponding to the holding hand includes: Parsing the reference relationship graph to determine a first distance between the object-holding hand and the original object; Based on a size ratio between the target object and the original object, resizing the first bounding box to obtain a third bounding box; Based on the distance, the position of the second bounding box is adjusted, wherein a second distance between the adjusted second bounding box and the third bounding box is equal to the first distance.

18. The method of claim 17, wherein: The method further comprises: The center point on the designated side of the third bounding box is moved to coincide with the center point on the designated side of the first bounding box.

19. A device for generating an interactive image, wherein: The device comprises: A first acquisition module is used to acquire a reference relationship graph, a target object graph and hand structure features; A first determination module, used to process the target object image to determine the appearance features of the target object; A second determination module, configured to process the reference relationship graph and the appearance feature to determine an interaction feature; A second acquisition module, used for fusing the interaction feature, the preset hand texture feature and the hand structure feature to obtain a first feature map; A third acquisition module, used for fusing the first feature map and a preset object texture feature map to obtain a second feature map; A generating module is used to generate an interaction graph including the target object based on the second feature graph.

20. The device of claim 19, wherein: The generating module is specifically used for: In a case where the reference relationship graph is an image frame in a reference video, the second feature graph is updated based on a third feature graph associated with other image frames in the reference video that has been output, to obtain an updated second feature graph; Based on the updated second feature map, an interaction map including the target object is generated.

21. The apparatus of claim 19, wherein: The first determining module is specifically configured to: Using a first encoder and a first convolutional network in sequence, encoding and convolution processing are performed on the target object image to obtain appearance features of the target object; The second determining module is specifically used to: Encoding the reference relationship graph using a second encoder to obtain the initial interaction feature; Using the i+1th convolutional layer in the second convolutional network, the interaction features output by the i+1th convolutional layer in the first convolutional network and the appearance features output by the i-th convolutional layer in the first convolutional network are convolved to obtain the i-th interaction feature, where i is a positive integer.

22. The device of claim 21, wherein: The second acquisition module is specifically used for: Using the i-th hand attention layer in the first attention network, the i-th interaction feature output by the i-th convolutional layer of the second convolutional network, the i-th hand structure feature output by the i-th network layer in the hand structure feature processing network, and the hand texture feature are fused to obtain a first feature map of the i-th dimension; The third acquisition module is specifically used for: Using the i-th object attention layer in the second attention network, the output of the i-th hand attention layer of the second convolutional network and the object texture features are fused to obtain a second feature map of the i-th dimension.

23. The device of claim 22, wherein: The second acquisition module is further used for: Using the first fusion layer of the i-th dimension in the first fusion network, the first feature map of the i-th dimension, the i-th interaction feature and the hand mask map of the i-th dimension are fused to obtain an updated first feature map of the i-th dimension.

24. The device of claim 23, wherein: The second acquisition module is further used for: Using the second fusion layer of the i-th dimension in the second fusion network, the second feature map of the i-th dimension, the updated first feature map of the i-th dimension, and the mask map corresponding to the target object are fused to obtain an updated second feature map of the i-th dimension.

25. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that may be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating an interactive image as described in any one of claims 10-18.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method for generating an interactive image according to any one of claims 10-18.

27. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method for generating an interactive image according to any one of claims 10 to 18.

Images