Image generation method, computer terminal, storage medium, and program product

Abstract

The invention discloses an image generation method, a computer terminal, a storage medium and a program product. The method comprises the steps that an input instruction acting on an operation interface is responded, a first image of a target object and second images of multiple target objects are displayed on the operation interface, and different target objects are used for being added to different positions of the target object; an image generation instruction acting on the operation interface is responded, a target image is displayed on the operation interface, a plurality of target objects are added to a target object in the target image, and the target image is an image generated by guiding the image generation model based on the object features of the target objects and the first image. The technical problem that the fidelity degree and the detail reduction degree of the virtual fitting image in a multi-piece clothing scene are low is solved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and image processing technology, and more specifically, to an image generation method, a computer terminal, a storage medium and a program product. Background Art

[0002] In e-commerce operations, when merchants put clothing products on the shelves, they usually need to hire models for trial wearing and shooting. This not only involves the cost of models and photography, but also requires repeated design in terms of posture, background, and light and shadow, making the new product launch process cumbersome and time-consuming. In order to solve this problem, virtual try-on technology came into being.

[0003] However, in the scenario of trying on multiple pieces of clothing, when generating virtual try-on images based on virtual try-on technology, it is difficult to achieve the mixed matching effect of tops and bottoms and the layered display of inner and outer clothing, and there are technical problems of low fidelity and detail restoration.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide an image generation method, a computer terminal, a storage medium, and a program product to at least solve the technical problem of low fidelity and detail restoration of virtual try-on images in a multi-piece clothing scene.

[0006] According to one aspect of an embodiment of the present application, there is provided an image generation method, comprising: in response to an input instruction acting on an operation interface, displaying a first image of a target object and second images of multiple target items on the operation interface, wherein different target items are used to be added at different positions of the target object; in response to an image generation instruction acting on the operation interface, displaying a target image on the operation interface, wherein multiple target items are added to the target object in the target image, the target image is an image generated by guiding an image generation model based on item features of the multiple target items and the first image, and the item features of the multiple target items are features obtained by encoding the second images of the multiple target items using an item coding model.

[0007] According to another aspect of an embodiment of the present application, an image generation method is also provided, including: acquiring a first image of a target object and a second image of multiple target items, wherein different target items are used to be added at different positions of the target object; encoding the second images of the multiple target items using an item coding model to obtain item features of the multiple target items; and guiding an image generation model to generate a target image based on the item features of the multiple target items and the first image, wherein the multiple target items are added to the target object in the target image.

[0008] According to another aspect of an embodiment of the present application, an image generation method is also provided, including: in response to an input instruction acting on an operation interface, displaying a first image of a biological object and a second image of multiple clothing items on the operation interface, wherein different clothing items are used to be worn on different parts of the biological object; in response to an image generation instruction acting on the operation interface, displaying a target image on the operation interface, wherein the biological object in the target image is wearing multiple clothing items, and the target image is an image generated based on clothing features of the multiple clothing items and an image generation model guided by the first image, and the clothing features of the multiple clothing items are features obtained by encoding the second image of the multiple clothing items using a clothing coding model.

[0009] According to another aspect of an embodiment of the present application, an image generation method is also provided, including: acquiring a first image of a target object and second images of multiple target items by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the first image and the second images of multiple target items, and different target items are used to be added to different positions of the target object; encoding the second images of the multiple target items using an object coding model to obtain object features of the multiple target items; based on the object features of the multiple target items and the first image, guiding an image generation model to generate a target image, wherein multiple target items are added to the target object in the target image; outputting the target image by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target image.

[0010] According to another aspect of the embodiments of the present application, a computer terminal is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the method in each embodiment of the present application is executed when the program is running.

[0011] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in each embodiment of the present application.

[0012] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0013] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0014] According to another aspect of the embodiments of the present application, a computer program is also provided, and when the computer program is executed by a processor, the methods in the various embodiments of the present application are implemented.

[0015] In an embodiment of the present application, in response to an input instruction acting on an operation interface, a first image of a target object and a second image of multiple target items are displayed on the operation interface, wherein different target items are used to be added at different positions of the target object; in response to an image generation instruction acting on the operation interface, the target image is displayed on the operation interface. It is easy to notice that the second image containing multiple target items is encoded using an item coding model to obtain item features of multiple target items, and based on the item features of multiple target items and the first image containing the target object, the image generation model is guided to generate the target image, and by introducing the fine-grained item features of multiple target items when generating the target image, the target image with multiple target items added to the target object can show more realistic fabric textures and natural lighting effects, accurately reproduce the details of the logo, pattern, text and stripes of the target item, and improve the robustness of the generated target image, thereby achieving the technical effect of improving the fidelity and detail restoration of the virtual try-on image, thereby solving the technical problem of low fidelity and detail restoration of the virtual try-on image in the scene of multiple clothing.

[0016] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a schematic diagram of an application scenario of an image generation method according to an embodiment of the present application;

[0019] Figure 2 is a structural block diagram of a computing environment of an image generation method according to an embodiment of the present application;

[0020] Figure 3 is a flowchart of an image generating method according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of an optional method of extracting item features using a feature extraction network according to an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of an optional method for generating image features according to an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of an optional method of using an image generation model to generate a target image according to an embodiment of the present application;

[0024] Figure 7 is a schematic diagram of self-attention processing performed by an optional second self-attention layer according to an embodiment of the present application;

[0025] Figure 8 is a schematic diagram of generating a second preset feature based on a multimodal processing model according to an optional embodiment of the present application;

[0026] Fig. 9 is a schematic diagram of an image generating device according to an embodiment of the present application;

[0027] Fig.10 is a flow chart of an image generating method according to an embodiment of the present application;

[0028] Fig.11 is a schematic diagram of an image generating device according to an embodiment of the present application;

[0029] Fig.12 is a flow chart of an image generating method according to an embodiment of the present application;

[0030] Fig.13 is a schematic diagram of an image generating device according to an embodiment of the present application;

[0031] Fig.14 is a flow chart of an image generating method according to an embodiment of the present application;

[0032] Fig.15 is a schematic diagram of an image generating device according to an embodiment of the present application;

[0033] Fig.16 It is a structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0037] SD: Stable Diffusion, diffusion model, is a generative model.

[0038] Unet network: A convolutional neural network with downsampling and upsampling modules, which is one of the basic building blocks of the Stable Diffusion model.

[0039] Tryon: Virtual try-on technology, which takes a portrait image and a flat target clothing image as input, removes the original clothing of the portrait, and integrates the target clothing image onto the portrait in a natural way.

[0040] HydraNet model: is a neural network model for multi-target detection and tracking, which performs well in handling complex multi-target detection and tracking tasks with high accuracy and robustness.

[0041] IP-Adapter: Image and text fusion adapter, used to combine images and text.

[0042] According to an embodiment of the present application, an image generation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] Considering that the machine learning model consumes a lot of computing resources of the mobile terminal, the above method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1In the application scenario shown, the machine learning model is deployed in the server 10, and the server 10 can be connected to one or more clients 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client 20 here may include but is not limited to: smart phones, tablet computers, laptops, PDAs, personal computers, smart home devices, vehicle-mounted devices, etc. The client 20 can interact with the user through a graphical user interface to call the large model, thereby implementing the method provided in the embodiment of the present application.

[0044] It should be noted that, when the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiments of the present application can be carried out in the client device.

[0045] Figure 2 is a structural block diagram of a computing environment of an image generation method according to an embodiment of the present application, such as Figure 2 As shown, computing environment 201 includes multiple computing nodes (such as servers) running on a distributed network (shown as 210-1, 210-2, ...) in the figure. Different computing nodes contain local processing and memory resources, and end users 202 can remotely run applications or store data in computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3 and 220-4 in computing environment 201, representing services "A", "D", "E" and "H" respectively.

[0046] The end user 202 can provide and access services through a web browser or other software application on the client, and in some embodiments, the end user 202's provision and / or request can be provided to the entry gateway 230. The entry gateway 230 can include a corresponding agent to handle the provision and / or request for the service (one or more services provided in the computing environment 201).

[0047] Services are provided or deployed based on various virtualization technologies supported by the computing environment 201. In some embodiments, services can be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can be to simulate a real computer by initializing a virtual machine, and execute programs and applications without directly contacting any actual hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, a container can be started to virtualize the entire operating system (OS) so that multiple workloads can run on a single operating system instance.

[0048] In an embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). Figure 2 As shown, service 220-2 can be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). Pods can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers in a Pod process requests related to one or more corresponding functions of the service, and the proxy 245 generally controls network functions related to the service, such as routing, load balancing, etc. Other services can also be equipped with similar Pods.

[0049] During operation, executing a user request from end user 202 may require invoking one or more services in computing environment 201, and executing one or more functions of one service may require invoking one or more functions of another service. Figure 2 As shown, service "A" 220-1 receives a user request from end user 202 from ingress gateway 230, service "A" 220-1 may call service "D" 220-2, and service "D" 220-2 may request service "E" 220-3 to perform one or more functions.

[0050] The computing environment described above can be a cloud computing environment, where the allocation of resources is managed by the cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be divided into a set of functions that can be automatically and independently scaled, rather than expanding a single hardware device to handle potential loads.

[0051] Under the above operating environment, this application provides Figure 3 It should be noted that the image generation method of this embodiment can be Figure 1 The server implementation of the illustrated embodiment. Figure 3 FIG. 1 is a flow chart of an image generation method according to an embodiment of the present application. Figure 3 As shown, the method may include the following steps:

[0052] Step S302, in response to an input instruction on the operation interface, a first image of the target object and second images of a plurality of target items are displayed on the operation interface, wherein different target items are used to be added at different positions of the target object.

[0053] The input instruction in the above steps is an instruction for the user to input the first image and the second image to be processed, which can be generated based on the interaction between the user and the operation interface. For example, the user can click the control for uploading the first image and the second image in the operation interface, and then upload the first image and multiple second images. After the client detects the user's operation of uploading the first image and multiple second images, it generates an input instruction, and the input instruction acts on the operation interface, instructing the operation interface to display the first image and multiple second images.

[0054] The target object in the above steps can be a living object such as a person or an animal, or an inanimate object such as a dining table, a TV wall, or a TV cabinet. In the embodiment of the present application, the target object is taken as a person for illustration.

[0055] The first image in the above step is an image containing the target object, for example, an image obtained by directly photographing the target object.

[0056] The target item in the above steps may be an item that you want to display on the target object. For example, assuming that the target object is a person, the target item may be clothing, shoes, accessories, etc. For another example, assuming that the target object is a dining table, the target item may be a tablecloth spread on the dining table, tableware placed on the dining table, etc. In the embodiment of the present application, the target item is taken as an example for explanation.

[0057] The second image in the above step is an image containing the target object. Different target objects may be contained in different second images. For example, multiple target objects may be photographed separately to obtain multiple second images.

[0058] In an optional embodiment, a virtual try-on scene in the context of e-commerce is used as an example for explanation. When a user needs to perform a virtual try-on based on a first image and multiple second images, the first image and multiple second images can be acquired first. When acquiring the first image, a camera can be used to shoot the target object to obtain the first image, or the first image containing the target object can be downloaded from the network or e-commerce platform, but the method of acquiring the first image may not be limited to this. For example, the target object may be a person, and the first image is an image containing the person. When acquiring multiple second images, multiple target items can be shot by mobile devices, cameras and other devices to obtain multiple second images, and multiple second images containing multiple target items can be downloaded from the network or e-commerce platform, but the method of acquiring multiple second images may not be limited to this. For example, the target object may be clothing, and the multiple second images may be images containing multiple different clothing.

[0059] Then, the user clicks on the control for uploading the first image and multiple second images in the operation interface to upload the first image and multiple second images. The purpose of the user uploading the first image and multiple second images is to expect that different clothes in the multiple second images can be displayed in different positions of the person, so that the user can observe the effect of the person in the first image trying on multiple clothes in the multiple second images. After the client detects the user's operation of uploading the first image and multiple second images, it generates an input instruction and instructs the operation interface to display the first image and multiple second images based on the input instruction.

[0060] Step S304, in response to the image generation instruction acting on the operation interface, a target image is displayed on the operation interface, wherein a plurality of target items are added to the target object in the target image, and the target image is an image generated by guiding the image generation model based on the item features of the plurality of target items and the first image, and the item features of the plurality of target items are features obtained by encoding the second images of the plurality of target items using the item coding model.

[0061] The image generation instruction in the above steps is an instruction for processing the first image and the plurality of second images to generate a target image, and can be generated based on the interaction between the user and the operation interface. Exemplarily, after the operation interface displays the first image and the plurality of second images, the user can click on the target image generation control in the operation interface, and after the client detects the user's click operation, it generates an image generation instruction, and the image generation instruction acts on the operation interface to instruct the operation interface to display the target image.

[0062] The target image in the above steps is generated based on the first image and the plurality of second images, and the target object is added with images of the plurality of target items. When the target object is a person and the target items are clothing, the user can understand the fitting effect of the person trying on multiple pieces of clothing at the same time through the target image.

[0063] The item features in the above steps are fine-grained coding features of the target item. When the target item is clothing, it can be a logo, pattern, text, stripes, etc. on the clothing, which can be obtained by encoding multiple second images through the item coding model. Generating a target image based on the item features of multiple target items and the first image can better show the details of multiple target items on the target object in the target image, thereby improving the authenticity of the try-on effect.

[0064] The object encoding model in the above steps is used to process multiple second images and extract the object features of multiple target objects. Optionally, it can be represented by HydraNet in the embodiment of the present application. The image generation model is used to process the target features of multiple objects and the first image to generate a target image. Optionally, it can be represented by MainNet in the embodiment of the present application.

[0065] In an optional embodiment, after the first image and the plurality of second images are displayed on the operation interface, the user may click on a target image generation control in the operation interface, and the client generates an image generation instruction based on the user's click operation and sends it to the server.

[0066] The server processes the first image showing a person and multiple second images showing different clothing based on the image generation instruction. During the processing, the multiple second images are input into the item coding model. Taking the item coding model as the HydraNet model as an example, the HydraNet model can handle multiple tasks at the same time. The HydraNet model processes the multiple second images and extracts the item features of multiple target items in the multiple second images. The item features of the multiple target items output by the HydraNet model are input into the image generation model together with the first image. Taking the image generation model as the MainNet model as an example, the MainNet model generates a target image showing multiple different clothing on the person based on the item features of the multiple target items and the first image.

[0067] The server returns the target image to the client, and the client instructs the operation interface to display the target image so that the user can observe the trying-on effects of the person trying on multiple different clothes.

[0068] In an embodiment of the present application, in response to an input instruction acting on an operation interface, a first image of a target object and a second image of multiple target items are displayed on the operation interface, wherein different target items are used to be added at different positions of the target object; in response to an image generation instruction acting on the operation interface, the target image is displayed on the operation interface. It is easy to notice that the second image containing multiple target items is encoded using an item coding model to obtain item features of multiple target items, and based on the item features of multiple target items and the first image containing the target object, the image generation model is guided to generate the target image, and by introducing the fine-grained item features of multiple target items when generating the target image, the target image with multiple target items added to the target object can show more realistic fabric textures and natural lighting effects, accurately reproduce the details of the logo, pattern, text and stripes of the target item, and improve the robustness of the generated target image, thereby achieving the technical effect of improving the fidelity and detail restoration of the virtual try-on image, thereby solving the technical problem of low fidelity and detail restoration of the virtual try-on image in the scene of multiple clothing.

[0069] In the above embodiment of the present application, the object coding model includes: multiple feature extraction networks, wherein the multiple feature extraction networks correspond one-to-one to the multiple target objects, the multiple feature extraction networks are executed in parallel, and the multiple feature extraction networks share some weights; the method also includes: inputting the second images of the multiple target objects into the corresponding feature extraction networks respectively, and obtaining the object features output by the corresponding feature extraction networks, so as to obtain the object features of the multiple target objects.

[0070] In an optional embodiment, the item coding model includes multiple feature extraction networks. The multiple feature extraction networks can work in parallel, simultaneously processing the second images containing multiple target items, and extracting the item features of the multiple target items. By parallelizing the multiple feature extraction networks, the item coding model can be allowed to expand the number of second images that can be processed, and the parameter increase of each additional branch is only 8%. Multiple feature extraction networks share some weights, which can reduce the number of parameters of the item coding model, reduce the complexity of the item coding model, and thus reduce the risk of overfitting.

[0071] When extracting item features of multiple target items, multiple second images are input into the item coding model, and each feature extraction network in the item coding model processes one second image to extract item features of one target item. Multiple feature extraction networks in the item coding model process in parallel to obtain item features of multiple target items at the same time.

[0072] Exemplarily, when the target item is clothing, multiple feature extraction networks can extract the coding features of multiple clothing in parallel, that is, the above-mentioned item features. When extracting the coding features of multiple clothing, multiple images containing clothing are input into the item coding model, and each feature extraction network in the item coding model processes an image containing clothing to extract a coding feature of a clothing. Multiple feature extraction networks in the item coding model process in parallel, and the coding features of multiple clothing can be obtained at the same time.

[0073] In the above embodiments of the present application, each feature extraction network includes: at least one feature extraction module, each feature extraction module includes a first self-attention layer and a first cross-attention layer, wherein multiple feature extraction networks share the weight of the first cross-attention layer; the second images of multiple target items are respectively input into the corresponding feature extraction networks, and the item features output by the corresponding feature extraction networks are obtained, including: using the first self-attention layer to perform self-attention processing on the input features of the feature extraction module to obtain a first self-attention feature, wherein, when the feature extraction module is the first feature extraction module in at least one feature extraction module, the input feature of the feature extraction module is the second image of the corresponding target item, and when the feature extraction module is not the first feature extraction module, the input feature of the feature extraction module is the output feature of the previous feature extraction module; using the first cross-attention layer to perform cross-attention processing on the first self-attention feature and the first preset feature to obtain the output feature of the feature extraction module; determining the output feature of the last feature extraction module in at least one feature extraction module as the item feature output by the corresponding feature extraction network.

[0074] The feature extraction module in the above steps is a component of the feature extraction network. Each feature extraction network includes one or more feature extraction modules, and the feature extraction module is used to extract the object features of the target object in the second image.

[0075] The first attention layer and the first cross-attention layer in the above steps are components of the feature extraction module, and the first attention layer is located before the first cross-attention layer. Optionally, in an embodiment of the present application, the first attention layer can be used for Hydra Encoding Block representation, used to extract information about input features and generate representations, and the first cross-attention layer can be a cross-attention layer, used to capture the relationship between input features and output features. The first attention layer is used for self-attention processing, and the self-attention processing is used to capture the correlation between different positions within the input sequence. The first cross-attention layer is used for cross-attention processing, and the cross-attention processing is used to capture the correlation between different inputs. The difference between self-attention processing and cross-attention processing is that the objects processed are different, one is an internal correlation relationship, and the other is an external correlation relationship. It should be noted that multiple feature extraction networks share the weights of the first cross-attention layer. By sharing weights, the number of parameters of the model can be reduced, the training burden of the item coding model can be reduced, and the generalization and promotion capabilities of the item coding model can be improved.

[0076] The input features in the above steps can be used as inputs of the feature extraction module. When the feature extraction module is the first feature extraction module, the input features are the second image. The output features of the previous feature extraction module can be used as input features of the next feature extraction module. Therefore, when the feature extraction module is not the first feature extraction module, the input features are the output features of the previous feature extraction module.

[0077] The output features in the above steps can be used as the output of the feature extraction module, and the output features of the last feature extraction module are the object features of the target object in the second image.

[0078] The first self-attention feature in the above step is the feature obtained by the first self-attention layer performing self-attention processing on the input feature, and can be used as the input of the first cross-attention layer in the same feature extraction module.

[0079] The first preset feature in the above step is a pre-set feature, which can be a text feature used to describe clothing, such as whether the clothes are open, whether the sleeves are rolled up, etc., or it can be a feature of an empty character, but is not limited thereto.

[0080] In an optional embodiment, each feature extraction network is used to extract an object feature of a target object in a second image. Taking the target object as clothing and the first preset feature as a text feature describing the clothing as an example, each feature extraction network includes one or more feature extraction modules, which can splice the second image containing the clothing and the text feature describing the clothing and input them into the feature extraction network.

[0081] The first self-attention layer of the first feature extraction module in the feature extraction network performs self-attention processing on the second image as the input feature to obtain the first self-attention feature, and then the first cross-attention layer in the first feature extraction module performs cross-attention processing on the first self-attention feature and the text feature describing the clothing to obtain the output feature of the first feature extraction module. The output feature of the first feature extraction module is used as the input feature of the second feature extraction module, and the first self-attention layer of the second feature extraction module performs self-attention processing on the input feature to obtain the first self-attention feature, and then the first cross-attention layer in the second feature extraction module performs cross-attention processing on the first self-attention feature and the text feature describing the clothing to obtain the output feature of the second feature extraction module, and so on, until the output feature of the last feature extraction module is obtained, and the output feature of the last feature extraction module is used as the item feature. It should be noted that multiple feature extraction networks share the weight of the first cross-attention layer. By sharing the weight, the number of parameters of the model can be reduced, the training burden of the item coding model can be reduced, and the generalization and promotion capabilities of the item coding model can be improved.

[0082] For example, Figure 4 is a schematic diagram of an optional method of extracting item features using a feature extraction network according to an embodiment of the present application, such as Figure 4 As shown, a feature extraction network 400 in the HydraNet model includes multiple feature extraction modules, each of which includes a first self-attention layer, namely, a Hydra Encoding Block, and a first cross-attention layer, wherein the Hydra Encoding Block in the feature extraction module is in front and the first cross-attention layer is in the back. The picture containing clothing as the second picture is spliced ​​with the text features used to describe the clothing and input into the feature extraction network 400. The Hydra Encoding Block in the first feature extraction module in the feature extraction network 400 performs self-attention processing on the picture containing clothing to obtain a first self-attention feature, and the first cross-attention layer in the first feature extraction module processes the first self-attention feature and the text features used to describe the clothing to obtain the output feature of the first feature extraction module. The output feature is processed by the next feature extraction module, the output feature is processed by the Hydra Encoding Block, and the output of the Hydra Encoding Block and the text features used to describe the clothing are processed by the first cross-attention layer to obtain the output feature of the feature extraction module. This is deduced until the output feature of the last feature extraction module is obtained and output as the item feature.

[0083] In the above embodiment of the present application, the method also includes: splicing the object features of multiple target objects to obtain splicing features; extracting features of the first image to obtain image features of the first image; inputting the splicing features and image features into the image generation model, and obtaining the target image output by the image generation model.

[0084] The image features in the above steps are features extracted from the first image. When the target object contained in the first image is a person, the image features can represent the body features and posture information of the person, and the background information of the first image.

[0085] In an optional embodiment, a splicing operation is performed on the item features of multiple target items. Specifically, the item features of each target item can be expanded, the multi-dimensional feature matrix can be expanded into a one-dimensional vector form, and the multiple vectors corresponding to the multiple item features can be spliced ​​to obtain the spliced ​​features in vector form.

[0086] The skeleton information or posture information of the person in the first image is extracted, and the background information other than the person in the first image is extracted at the same time, so as to obtain a first image that can describe the person's posture information and the background information.

[0087] The splicing features and image features are input into the MainNet model, that is, the image generation model. The image generation model integrates the character posture information, background information and the splicing features of multiple costumes that need to be added to the character to generate a target image with multiple costumes displayed on the character.

[0088] In the above embodiment of the present application, the object features of multiple target objects are spliced ​​to obtain a spliced ​​feature, including: superimposing the object features of multiple target objects and corresponding position features to obtain multiple superimposed features, wherein different target objects have different corresponding position features; and splicing the multiple superimposed features to obtain a spliced ​​feature.

[0089] The superimposed features in the above steps are obtained by superimposing the position features on the basis of the object features. By superimposing the position features, the position of the target object on the target object can be marked, so that the target object can be added to the correct position on the target object when generating the target image.

[0090] The position features in the above steps can characterize the relative position of the target object on the target object. For example, taking the target object as a person, the position feature corresponding to the person's head is 0, the position code corresponding to the person's upper body is 1, the position code corresponding to the person's lower body is 2, the position code corresponding to the person's feet is 3, and so on.

[0091] In an optional embodiment, the position feature of the target object can be determined based on the relative position of the target object on the model in the second image. For example, when the target object in the second image is clothing and the target object in the first image is a person, the position feature is determined based on the position of the clothing on the model in the second image, such as determining the position feature of the top as being located on the upper body of the person, determining the position feature of the pants as being located on the lower body of the person, determining the position feature of the shoes as being located on the feet of the person, etc.

[0092] For each target object, the position feature of the target object is converted into a position code. On the basis of the object feature, the position code corresponding to the target object is superimposed to obtain the superimposed feature, thereby distinguishing the positions of different clothing.

[0093] The superimposed features of multiple target objects are converted into vector form, and the superimposed features in vector form are spliced ​​to obtain spliced ​​features.

[0094] In the above-mentioned embodiment of the present application, feature extraction is performed on the first image to obtain image features of the first image, including: extracting skeleton information of the target object in the first image to obtain a skeleton image of the target object, and performing feature extraction on the skeleton image to obtain skeleton features of the target object; and performing feature extraction on a mask image of the first image to obtain mask features of the first image, wherein the mask image is an image determined in response to an input instruction, or an image obtained by processing the first image based on the skeleton image; based on the position of the original object in the first image, the first image is processed to obtain a third image, and feature extraction is performed on the third image to obtain background features of the first image, wherein the original object is not included in the third image; and the skeleton features, mask features and background features are spliced ​​to obtain image features.

[0095] The skeleton information in the above steps is the main outline and structural information of the target object, which is usually used to characterize the shape and posture of the target object. The skeleton information can be obtained by extracting the edges or key points of the target object.

[0096] The skeleton image in the above step is an image drawn based on the skeleton information of the target object, and usually contains the main outline and structure of the target object, but does not contain specific texture or color information.

[0097] The skeleton features in the above steps refer to features extracted from the skeleton image, which may include shape information such as skeleton length, width, curvature, etc., and are used to describe the morphological characteristics of the target object.

[0098] The mask image in the above step is used to cover the clothes originally worn by the model in the first image. The mask image can be a binary image, in which the pixel value of the area where the model originally wore clothes is 0, indicating the area to be ignored, and the pixel value of other areas is 1, indicating the area to be retained.

[0099] The mask feature in the above step refers to feature information extracted from the mask image, which may include but is not limited to the shape, size, edge information, etc. of the mask area.

[0100] The third image in the above step is an image that does not contain the original object and is used to represent the background information of the first image. When the target object in the first image is a person, the third image can be a clothes-free image, which is a picture obtained by graying out the clothes area worn by the model in the first image.

[0101] In an optional embodiment, when the target object is a person, the skeleton information of the person in the first image can be extracted by dwpose (DeepWire PoseEstimation) technology. By using dwpose technology, the information of the human skeleton, including joint positions and connection relationships, etc., can be extracted from the first image to construct a skeleton image. Then, the skeleton image is converted into a feature vector by using Pose Encoder, and the extracted skeleton image is encoded into a vector of a fixed length to obtain skeleton features.

[0102] The mask image of the first image can be obtained by uploading a processed mask image by a user, or by automatically covering the whole body of the model in the first image with a skeleton image to obtain the mask image. After the mask image is obtained, the shape and structural features of the mask image can be extracted through edge detection to obtain the mask features.

[0103] When obtaining the clothing-independent image, i.e., the third image, the mask image can be used to process the first image, and the mask area in the mask image can be mapped to the first image to obtain the area to be processed, and the area to be processed can be filled with gray, and the background area can be kept consistent with the original first image, thereby obtaining the third image. Then, the third object can be processed by the trained feature extraction model to obtain background features.

[0104] The skeleton features, mask features and background features are converted into vector features and concatenated to obtain image features.

[0105] For example, Figure 5 is a schematic diagram of an optional method for generating image features according to an embodiment of the present application, such as Figure 5 As shown, the skeleton image 50 is input into the pose encoder, i.e., the Pose Encoder, to extract features and obtain skeleton features. At the same time, the mask image 52 and the clothing-independent image 54 are respectively extracted to obtain mask features and background features. The skeleton features, mask features, and background features are concatenated (i.e., c in the figure) to obtain image features 56, which can be input into the image generation model for generating the target image.

[0106] In the above-mentioned embodiment of the present application, the image generation model includes: at least one image generation module, each image generation module includes a second self-attention layer and a second cross-attention layer; the splicing features and the image features are input into the image generation model, and the target image output by the image generation model is obtained, including: using the second self-attention layer to perform self-attention processing on the input features and the splicing features of the image generation module to obtain the second self-attention features, wherein, when the image generation module is the first image generation module in at least one image generation module, the input features of the image generation module are the image features, and when the image generation module is not the first image generation module, the input features of the image generation module are the output features of the previous image generation module; using the second cross-attention layer to perform cross-attention processing on the second self-attention features and the second preset features to obtain the output features of the image generation module; based on the output features of the last image generation module in at least one image generation module, the target image is generated.

[0107] The image generation module in the above steps is a component of the image generation model. Each image generation model includes one or more image generation modules. The image generation module is used to generate a target image based on the splicing features and the image features.

[0108] The second self-attention layer and the second cross-attention layer in the above steps are components of the image generation module, and the second self-attention layer is located before the second cross-attention layer. Optionally, in an embodiment of the present application, the second self-attention layer can be used for Hydra Fusion Block representation, for fusing image features and splicing features, and the second cross-attention layer can be a cross-attention layer, for introducing a second preset feature for cross-attention processing based on fusing image features and splicing features.

[0109] The input features in the above steps can be used as inputs of the image generation module. When the image generation module is the first image generation module, the input features are image features. The output features of the previous image generation module can be used as input features of the next image generation module. Therefore, when the image generation module is not the first image generation module, the input features are the output features of the previous image generation module.

[0110] The output features in the above steps can be used as the output of the image generation module, and the output features of the last image generation module can be used to generate the target image.

[0111] The second self-attention feature in the above step is a feature obtained by the second self-attention layer performing self-attention processing on the input feature and the splicing feature, and can be used as the input of the second cross-attention layer in the same image generation module.

[0112] The second preset feature in the above step is a pre-set feature, which can be obtained by splicing the second image and the text feature used to describe the clothing, and is a high-level feature of the second image and the text feature.

[0113] In an optional embodiment, Figure 6 FIG. 1 is a schematic diagram of an optional method of using an image generation model to generate a target image according to an embodiment of the present application. Figure 6 As shown, the image generation model 600 includes multiple image generation modules, each of which includes a second self-attention layer, namely, Hydra Fusion Block and a second cross-attention layer. The image feature is used as the input feature of the first image generation module, and the Hydra FusionBlock in the first image generation module performs self-attention processing on the image feature and the splicing feature obtained by the object coding model to obtain the second self-attention feature. Then, the second cross-attention layer in the first image generation module performs cross-attention processing on the second self-attention feature and the second preset feature output by the IP-Adapter to obtain the output feature of the first image generation module. Among them, the input of the IP-Adapter is a second image containing clothing and text features for describing clothing. The IP-Adapter can output high-level features obtained by splicing the second image and the text features for describing clothing, namely, the second preset features. The output features of the first image generation module are used as the input features of the second image generation module. The Hydra Fusion Block of the second image generation module performs self-attention processing on the input features and the splicing features obtained by the item coding model to obtain the second self-attention features. Then, the second cross-attention layer in the second image generation module performs cross-attention processing on the second self-attention features and the second preset features output by the IP-Adapter to obtain the output features of the second image generation module, and so on, until the output features of the last image generation module are obtained. The output features of the last image generation module are input into the processing network, and the output features are decoded in the processing network to construct the target image. Among them, the processing network can be a GAN (Generative Adversarial Network) or a VAE (Variational Auto Encoder), but is not limited to this.

[0114] In the above embodiment of the present application, the second self-attention layer is used to perform self-attention processing on the input features and splicing features of the image generation module to obtain the second self-attention feature, including: determining the query vector corresponding to the second self-attention layer based on the input features of the image generation module; splicing the input features and splicing features of the image generation module to obtain the key vector and value vector corresponding to the second self-attention layer; using the query vector, key vector and value vector to obtain the second self-attention feature.

[0115] The query vector in the above steps is a vector used to calculate the attention weight, which can represent the information of the current position or the current time step, and is used to compare with the information of other positions or time steps to determine the importance of different positions or time steps.

[0116] The key vector in the above steps is also the vector used to calculate the attention weight, which can represent information of other positions or other time steps besides the current position, and is used to compare with the query vector to determine the importance of different positions or time steps.

[0117] The value vector in the above steps is the output vector obtained after attention weight calculation.

[0118] In an optional embodiment, Figure 7 is a schematic diagram of an optional second self-attention layer performing self-attention processing according to an embodiment of the present application, such as Figure 7 As shown, the second self-attention layer 700, i.e., Hydra Fusion Block, is used for self-attention processing, and the entire self-attention processing process is performed based on the input features. The image features can be used as the input features of the first image generation module. Based on the image features, the query vector in the second self-attention layer 700, i.e., Q in the figure, can be determined. Then, the input features 702 and the splicing features 704 of other image generation modules are spliced ​​to obtain the key vector and the value vector, i.e., K and V in the image. The query vector Q, the key vector K, and the value vector V are self-attentively processed to obtain the second self-attention feature.

[0119] In the above embodiment of the present application, the method also includes: obtaining a target text, wherein the target text is used to describe a target image; splicing the second images of multiple target objects to obtain a spliced ​​image; and performing feature extraction on the target text and the target image using a multimodal processing model to obtain a second preset feature, wherein the second preset feature is used to characterize the object information of the multiple target objects and the method of adding the multiple target objects to the target object.

[0120] The target text is a text used to describe the target image in the first image. For example, the target text may be that the sleeves of the clothes need to be rolled up, the clothes need to be buttoned, there are multiple blocks of different colors in the jeans, etc. The target text can provide some more detailed information on how to wear and improve the authenticity of the try-on effect.

[0121] The multimodal processing model in the above steps can be used to generate a second preset feature based on the target text and the spliced ​​image, and by generating the second preset feature, high-level features of the spliced ​​clothing image and text description are obtained. IP-Adapter can be selected as a multimodal processing model, but is not limited thereto.

[0122] The item information in the above steps may be relatively rough clothing information, such as clothing structure, color, etc.

[0123] The adding method in the above steps can be information about how the clothing is worn, such as whether the clothing is open, whether the sleeves are rolled up, etc.

[0124] In an optional embodiment, Figure 8 is a schematic diagram of generating a second preset feature based on a multimodal processing model according to an optional embodiment of the present application, such as Figure 8 As shown, a target text describing a target image is obtained. The target text can be input by the user. For example, the content of the target text can be that the model is wearing a black T-shirt and trousers with color blocks. Then, multiple images 800 showing clothing, that is, the second image, are spliced ​​to obtain a spliced ​​image. The target text and the spliced ​​image are input into the image-text fusion adapter, that is, the multimodal processing model, and the features of the target text and the spliced ​​image are extracted to obtain a second preset feature that can characterize the item information and the adding method.

[0125] In the above-mentioned embodiments of the present application, the initial network parameters of the image generation model are obtained by fusing the network parameters of multiple preset models, and the network structures of different preset models are the same but the network parameters are different.

[0126] The preset model in the above steps is a pre-set model, which can be preset according to the actual application situation, and the specific structure and parameters of the preset model are not limited here. For example, the SDXL-base-1.0 model, the SDXL-inpainting-0.1 model and the DreamshaperXL alpha2 model can be selected as preset models, wherein the above three models have the same network structure and different network parameters, and by fusing the network parameters of the above three models, the initial network parameters of the image generation model can be obtained.

[0127] The initial network parameters in the above steps are the initial values ​​of parameters such as weights and biases of the image generation model.

[0128] The network parameters of the preset model in the above steps are the values ​​of parameters such as weights and biases in the preset model.

[0129] In an optional embodiment, a priori model evolution strategy is adopted, different preset models are selected, and multiple models with the same network structure and different parameters are fused to obtain the initial network parameters of the image generation model. Different preset models have different characteristics, for example, some models are better at general image generation, and some models are better at generating models. By fusing the network parameters of multiple preset models, the image generation model can have the advantages of different preset models at the same time without training, so as to improve the quality and robustness of the target image generated by the image generation model.

[0130] In the above embodiment of the present application, the method also includes: determining fusion weights corresponding to multiple preset models; fusing network parameters of multiple preset models based on the fusion weights corresponding to the multiple preset models to obtain initial network parameters.

[0131] The fusion weight in the above steps can represent the contribution of the network parameters of different preset models to the generated initial network parameters during the fusion process.

[0132] In an optional embodiment, SDXL-base-1.0, SDXL-inpainting-0.1 models and DreamshaperXL alpha2 (are selected as preset models. Then, based on the results of multiple experiments, the value of the fusion weight α is determined to be 1 and the value of β is determined to be 1.1. Then, the network parameters and fusion weights of the above three preset models are fused, and the fusion formula is:

[0133] W new =W base +α·(W inp -W base )+β·(W ds -W base );

[0134] Among them, W new is the initial network parameter, W base is the network parameter of SDXL-base-1.0, W inp is the network parameter of SDXL-inpainting-0.1, W ds is the network parameter of DreamshaperXL alpha2, α and β are fusion weights, α is 1, and β is 1.1. inp With W baseThe difference is the increment of SDXL-inpainting-0.1 model relative to SDXL-base-1.0 model, W ds With W base The difference is the increment of DreamshaperXL alpha2 model relative to SDXL-base-1.0 model.

[0135] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0136] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0137] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including several instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0138] According to an embodiment of the present application, an image generating device for implementing the above-mentioned image generating method is also provided. Fig. 9 is a schematic diagram of an image generating device according to an embodiment of the present application, such as Fig. 9 As shown, the device comprises:

[0139] The first display module 90 is used to respond to an input instruction on the operation interface and display a first image of the target object and second images of multiple target items on the operation interface, wherein different target items are used to be added at different positions of the target object.

[0140] The second display module 92 responds to the image generation instruction acting on the operation interface and displays the target image on the operation interface, wherein a plurality of target items are added to the target object in the target image, and the target image is an image generated by guiding the image generation model based on the item features of the plurality of target items and the first image, and the item features of the plurality of target items are features obtained by encoding the second images of the plurality of target items using the item coding model.

[0141] In the above embodiment of the present application, the object coding model in the second display module includes: multiple feature extraction networks, wherein the multiple feature extraction networks correspond to the multiple target objects one by one, the multiple feature extraction networks are executed in parallel, and the multiple feature extraction networks share some weights. The second display module also includes: a first acquisition unit, which is used to input the second images of the multiple target objects into the corresponding feature extraction networks respectively, and obtain the object features output by the corresponding feature extraction networks to obtain the object features of the multiple target objects.

[0142] In the above embodiment of the present application, each feature extraction network in the first acquisition unit includes: at least one feature extraction module, each feature extraction module includes a first self-attention layer and a first cross-attention layer, wherein the weight of the first cross-attention layer is shared by multiple feature extraction networks. The first acquisition unit is also used to input the second images of multiple target items into the corresponding feature extraction network respectively, and obtain the item features output by the corresponding feature extraction network, including: using the first self-attention layer to perform self-attention processing on the input features of the feature extraction module to obtain a first self-attention feature, wherein, when the feature extraction module is the first feature extraction module in at least one feature extraction module, the input feature of the feature extraction module is the second image of the corresponding target item, and when the feature extraction module is not the first feature extraction module, the input feature of the feature extraction module is the output feature of the previous feature extraction module; using the first cross-attention layer to perform cross-attention processing on the first self-attention feature and the first preset feature to obtain the output feature of the feature extraction module; determining the output feature of the last feature extraction module in at least one feature extraction module as the item feature output by the corresponding feature extraction network.

[0143] In the above embodiment of the present application, the second display module also includes: a splicing unit, used to splice the object features of multiple target objects to obtain splicing features; a feature extraction unit, used to extract features from the first image to obtain image features of the first image; a second acquisition unit, used to input the splicing features and image features into the image generation model, and obtain the target image output by the image generation model.

[0144] In the above embodiments of the present application, the splicing unit is also used to superimpose the object features and corresponding position features of multiple target objects to obtain multiple superimposed features, wherein different target objects have different corresponding position features; and to splice the multiple superimposed features to obtain a spliced ​​feature.

[0145] In the above embodiment of the present application, the feature extraction unit is also used to extract the skeleton information of the target object in the first image to obtain the skeleton image of the target object, and perform feature extraction on the skeleton image to obtain the skeleton features of the target object; and perform feature extraction on the mask image of the first image to obtain the mask features of the first image, wherein the mask image is an image determined in response to an input instruction, or an image obtained by processing the first image based on the skeleton image; based on the position of the original object in the first image, the first image is processed to obtain a third image, and feature extraction is performed on the third image to obtain background features of the first image, wherein the original object is not included in the third image; the skeleton features, mask features and background features are spliced ​​to obtain image features.

[0146] In the above embodiment of the present application, the image generation model in the second acquisition unit includes: at least one image generation module, each image generation module includes a second self-attention layer and a second cross-attention layer. The second acquisition unit is also used to use the second self-attention layer to perform self-attention processing on the input features and splicing features of the image generation module to obtain a second self-attention feature, wherein, when the image generation module is the first image generation module in at least one image generation module, the input feature of the image generation module is the image feature, and when the image generation module is not the first image generation module, the input feature of the image generation module is the output feature of the previous image generation module; the second self-attention feature and the second preset feature are cross-attended using the second cross-attention layer to obtain the output feature of the image generation module; based on the output feature of the last image generation module in at least one image generation module, the target image is generated.

[0147] In the above embodiment of the present application, the second acquisition unit is also used to determine the query vector corresponding to the second self-attention layer based on the input features of the image generation module; splice the input features and splicing features of the image generation module to obtain the key vector and value vector corresponding to the second self-attention layer; and use the query vector, key vector and value vector to obtain the second self-attention feature.

[0148] In the above embodiment of the present application, the second acquisition unit is also used to acquire a target text, wherein the target text is used to describe a target image; the second images of multiple target objects are spliced ​​to obtain a spliced ​​image; and feature extraction is performed on the target text and the target image using a multimodal processing model to obtain a second preset feature, wherein the second preset feature is used to characterize the object information of the multiple target objects and the method of adding the multiple target objects to the target object.

[0149] In the above-mentioned embodiment of the present application, the initial network parameters of the image generation model in the second display module are obtained by fusing the network parameters of multiple preset models, and the network structures of different preset models are the same but the network parameters are different.

[0150] In the above embodiment of the present application, the second display module is also used to determine the fusion weights corresponding to multiple preset models; based on the fusion weights corresponding to the multiple preset models, the network parameters of the multiple preset models are fused to obtain the initial network parameters.

[0151] It should be noted here that the above-mentioned first display module 90 and second display module 92 correspond to steps S302 to S304 in the above-mentioned embodiments, and the two modules are the same as the instances and application scenarios implemented by the corresponding steps, but are not limited to the contents disclosed in the above-mentioned embodiments. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of the device and can be run in the AR / VR device provided in the above-mentioned embodiments. It should be noted that the preferred implementation scheme involved in the above-mentioned embodiments of the present application is the same as the scheme, application scenario, and implementation process provided in the above-mentioned embodiments, but is not limited to the scheme provided in the above-mentioned embodiments.

[0152] According to an embodiment of the present application, an embodiment of an image generating method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0153] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the above embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the above embodiments.

[0154] Fig.10 is a flow chart of an image generation method according to an embodiment of the present application, such as Fig.10 As shown, the method may include the following steps:

[0155] Step S1002, acquiring a first image of a target object and second images of a plurality of target objects, wherein different target objects are used to be added at different positions of the target object;

[0156] Step S1004, encoding the second images of the plurality of target objects using the object encoding model to obtain object features of the plurality of target objects;

[0157] Step S1006: Based on the item features of the plurality of target items and the first image, guide the image generation model to generate a target image, wherein the plurality of target items are added to the target object in the target image.

[0158] According to an embodiment of the present application, an image generating device for implementing the above-mentioned image generating method is also provided. Fig.11 is a schematic diagram of an image generating device according to an embodiment of the present application, such as Fig.11 As shown, the device comprises:

[0159] The acquisition module 110 is used to acquire a first image of the target object and second images of a plurality of target objects, wherein different target objects are used to be added at different positions of the target object.

[0160] The encoding module 112 is used to encode the second images of the multiple target objects using the object encoding model to obtain the object features of the multiple target objects.

[0161] The generation module 114 is used to guide the image generation model to generate a target image based on the item features of the multiple target items and the first image, wherein the multiple target items are added to the target object in the target image.

[0162] It should be noted here that the above-mentioned acquisition module 110, encoding module 112 and generation module 114 correspond to steps S1002 to S1006 in the above-mentioned embodiments, and the three modules are the same as the instances and application scenarios implemented by the corresponding steps, but are not limited to the contents disclosed in the above-mentioned embodiments. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of the device and can be run in the AR / VR device provided in the above-mentioned embodiments. It should be noted that the preferred implementation scheme involved in the above-mentioned embodiments of the present application is the same as the scheme, application scenario and implementation process provided in the above-mentioned embodiments, but is not limited to the scheme provided in the above-mentioned embodiments.

[0163] According to an embodiment of the present application, an embodiment of an image generating method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0164] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the above embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the above embodiments.

[0165] Fig.12 is a flow chart of an image generation method according to an embodiment of the present application, such as Fig.12 As shown, the method may include the following steps:

[0166] Step S1202, in response to an input instruction on the operation interface, displaying a first image of the biological object and second images of a plurality of clothing items on the operation interface, wherein different clothing items are used to be worn on different parts of the biological object;

[0167] Step S1204, in response to the image generation instruction acting on the operation interface, a target image is displayed on the operation interface, wherein the biological object in the target image wears multiple pieces of clothing, and the target image is an image generated based on clothing features of the multiple pieces of clothing and a first image-guided image generation model, and the clothing features of the multiple pieces of clothing are features obtained by encoding the second image of the multiple pieces of clothing using a clothing coding model.

[0168] The biological object in the above steps is a living organism with life characteristics, which can be a human or an animal.

[0169] The clothing features in the above steps are the characteristic expressions of clothing in the image, which may include but are not limited to the characteristics of clothing in terms of color, texture, style, matching, etc. By encoding the second image containing clothing, the characteristics of clothing can be extracted, so that the clothing worn on the biological object can be accurately expressed in the process of generating the target image.

[0170] According to an embodiment of the present application, an image generating device for implementing the above-mentioned image generating method is also provided. Fig.13 is a schematic diagram of an image generating device according to an embodiment of the present application, such as Fig.13 As shown, the device comprises:

[0171] The first display module 130 is used to respond to an input instruction on the operation interface and display a first image of the biological object and second images of multiple clothing items on the operation interface, wherein different clothing items are used to be worn on different parts of the biological object.

[0172] The second display module 132 is used to respond to the image generation instruction acting on the operation interface and display the target image on the operation interface, wherein the biological object in the target image wears multiple pieces of clothing, and the target image is an image generated based on clothing features of the multiple pieces of clothing and the first image guided image generation model, and the clothing features of the multiple pieces of clothing are features obtained by encoding the second image of the multiple pieces of clothing using the clothing coding model.

[0173] It should be noted here that the above-mentioned first display module 130 and the second display module 132 correspond to steps S1202 to S1204 in the above-mentioned embodiments, and the two modules are the same as the instances and application scenarios implemented by the corresponding steps, but are not limited to the contents disclosed in the above-mentioned embodiments. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of the device and can be run in the AR / VR device provided in the above-mentioned embodiments. It should be noted that the preferred implementation scheme involved in the above-mentioned embodiments of the present application is the same as the scheme, application scenario, and implementation process provided in the above-mentioned embodiments, but is not limited to the scheme provided in the above-mentioned embodiments.

[0174] According to an embodiment of the present application, an embodiment of an image generating method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0175] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the above embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the above embodiments.

[0176] Fig.14 is a flow chart of an image generation method according to an embodiment of the present application, such as Fig.12 As shown, the method may include the following steps:

[0177] Step S1402, acquiring a first image of the target object and second images of multiple target objects by calling a first interface, wherein the first interface includes a first parameter, a parameter value of the first parameter includes the first image and the second images of multiple target objects, and different target objects are used to be added to different positions of the target object;

[0178] Step S1404, encoding the second images of the plurality of target objects using the object encoding model to obtain object features of the plurality of target objects;

[0179] Step S1406, based on the item features of the plurality of target items and the first image, guiding the image generation model to generate a target image, wherein the plurality of target items are added to the target object in the target image;

[0180] Step S1408: output the target image by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target image.

[0181] The first interface in the above steps is an interface for acquiring the first image and the plurality of second images, the first parameter is a parameter of the first interface, and the parameter value may be the first image and the plurality of second images.

[0182] The second interface in the above steps is an interface for outputting a target image, the second parameter is a parameter of the second interface, and the parameter value may be the target image.

[0183] According to an embodiment of the present application, an image generating device for implementing the above-mentioned image generating method is also provided. Fig.15 is a schematic diagram of an image generating device according to an embodiment of the present application, such as Fig.15 As shown, the device comprises:

[0184] The acquisition module 150 is used to acquire a first image of the target object and second images of multiple target objects by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the first image and the second images of multiple target objects, and different target objects are used to be added at different positions of the target object.

[0185] The encoding module 152 is used to encode the second images of the multiple target objects using the object encoding model to obtain the object features of the multiple target objects.

[0186] The generation module 154 is used to guide the image generation model to generate a target image based on the item features of the multiple target items and the first image, wherein the multiple target items are added to the target object in the target image.

[0187] The output module 156 is configured to output the target image by calling the second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter includes the target image.

[0188] It should be noted here that the above-mentioned acquisition module 150, encoding module 152, generation module 154 and output module 156 correspond to steps S1402 to S1408 in the above-mentioned embodiments, and the four modules are the same as the instances and application scenarios implemented by the corresponding steps, but are not limited to the contents disclosed in the above-mentioned embodiments. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of the device and can run in the AR / VR device provided in the above-mentioned embodiments. It should be noted that the preferred implementation scheme involved in the above-mentioned embodiments of the present application is the same as the scheme, application scenario and implementation process provided in the above-mentioned embodiments, but is not limited to the scheme provided in the above-mentioned embodiments.

[0189] An embodiment of the present application may provide a computer terminal, which may include a server and a client. The server may be any server in a server device group or a cloud server.

[0190] In the above embodiment of the present application, in this embodiment, the above computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0191] In this embodiment, the computer terminal can execute the program code in the method.

[0192] In the above embodiments of the present application, Fig.16 It is a structural block diagram of a computer terminal according to an embodiment of the present application. As shown in the figure, the computer terminal A may include: one or more (only one is shown in the figure) processors 1602, a memory 1604, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0193] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0194] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: responding to the input instructions acting on the operation interface, displaying the first image of the target object and the second images of multiple target items on the operation interface, wherein different target items are used to be added at different positions of the target object; responding to the image generation instructions acting on the operation interface, displaying the target image on the operation interface, wherein multiple target items are added to the target object in the target image, and the target image is an image generated by guiding the image generation model based on the item features of the multiple target items and the first image, and the item features of the multiple target items are features obtained by encoding the second images of the multiple target items using the item coding model.

[0195] Optionally, the processor may also execute the following program code: the item coding model includes: a plurality of feature extraction networks, wherein the plurality of feature extraction networks correspond to the plurality of target items one by one, the plurality of feature extraction networks are executed in parallel, and the plurality of feature extraction networks share some weights. The method also includes: inputting the second images of the plurality of target items into the corresponding feature extraction networks respectively, and obtaining the item features output by the corresponding feature extraction networks, to obtain the item features of the plurality of target items.

[0196] Optionally, the processor may also execute the following steps of program code: each feature extraction network includes: at least one feature extraction module, each feature extraction module includes a first self-attention layer and a first cross-attention layer, wherein multiple feature extraction networks share the weight of the first cross-attention layer; the second images of multiple target items are respectively input into the corresponding feature extraction networks, and the item features output by the corresponding feature extraction networks are obtained, including: using the first self-attention layer to perform self-attention processing on the input features of the feature extraction module to obtain a first self-attention feature, wherein, when the feature extraction module is the first feature extraction module in at least one feature extraction module, the input feature of the feature extraction module is the second image of the corresponding target item, and when the feature extraction module is not the first feature extraction module, the input feature of the feature extraction module is the output feature of the previous feature extraction module; using the first cross-attention layer to perform cross-attention processing on the first self-attention feature and the first preset feature to obtain the output feature of the feature extraction module; determining the output feature of the last feature extraction module in at least one feature extraction module as the item feature output by the corresponding feature extraction network.

[0197] Optionally, the processor may also execute the program code of the following steps: the method also includes: splicing the object features of multiple target objects to obtain splicing features; extracting features of the first image to obtain image features of the first image; inputting the splicing features and image features into the image generation model, and obtaining the target image output by the image generation model.

[0198] Optionally, the processor may also execute program code for the following steps: splicing item features of multiple target items to obtain a spliced ​​feature, including: superimposing item features and corresponding position features of multiple target items to obtain multiple superimposed features, wherein different target items have different corresponding position features; and splicing multiple superimposed features to obtain a spliced ​​feature.

[0199] Optionally, the processor may also execute program code for the following steps: extracting features from the first image to obtain image features of the first image, including: extracting skeleton information of the target object in the first image to obtain a skeleton image of the target object, and extracting features from the skeleton image to obtain skeleton features of the target object; and extracting features from a mask image of the first image to obtain mask features of the first image, wherein the mask image is an image determined in response to an input instruction, or an image obtained by processing the first image based on the skeleton image; processing the first image to obtain a third image based on the position of the original object in the first image, and extracting features from the third image to obtain background features of the first image, wherein the original object is not included in the third image; and splicing the skeleton features, mask features, and background features to obtain image features.

[0200] Optionally, the processor may also execute the program code of the following steps: the image generation model includes: at least one image generation module, each image generation module includes a second self-attention layer and a second cross-attention layer; the splicing features and the image features are input into the image generation model, and the target image output by the image generation model is obtained, including: using the second self-attention layer to perform self-attention processing on the input features and the splicing features of the image generation module to obtain the second self-attention features, wherein, when the image generation module is the first image generation module in at least one image generation module, the input features of the image generation module are the image features, and when the image generation module is not the first image generation module, the input features of the image generation module are the output features of the previous image generation module; using the second cross-attention layer to perform cross-attention processing on the second self-attention features and the second preset features to obtain the output features of the image generation module; generating the target image based on the output features of the last image generation module in at least one image generation module.

[0201] Optionally, the processor may also execute the program code of the following steps: using the second self-attention layer to perform self-attention processing on the input features and splicing features of the image generation module to obtain a second self-attention feature, including: determining a query vector corresponding to the second self-attention layer based on the input features of the image generation module; splicing the input features and splicing features of the image generation module to obtain a key vector and a value vector corresponding to the second self-attention layer; and using the query vector, the key vector and the value vector to obtain the second self-attention feature.

[0202] Optionally, the processor may also execute the program code of the following steps: the method also includes: obtaining a target text, wherein the target text is used to describe a target image; splicing the second images of multiple target objects to obtain a spliced ​​image; and performing feature extraction on the target text and the target image using a multimodal processing model to obtain a second preset feature, wherein the second preset feature is used to characterize the object information of the multiple target objects and the method of adding the multiple target objects to the target object.

[0203] Optionally, the processor may also execute program code of the following steps: initial network parameters of the image generation model are obtained by fusing network parameters of multiple preset models, and different preset models have the same network structure but different network parameters.

[0204] Optionally, the processor may also execute the program code of the following steps: the method also includes: determining fusion weights corresponding to multiple preset models; fusing network parameters of multiple preset models based on the fusion weights corresponding to the multiple preset models to obtain initial network parameters.

[0205] By adopting the embodiment of the present application, a scheme of an image generation method is provided. The second image containing multiple target items is encoded using an item coding model to obtain item features of the multiple target items, and based on the item features of the multiple target items and the first image containing the target object, the image generation model is guided to generate the target image, and the fine-grained item features of the multiple target items are introduced when generating the target image, so that the target image with multiple target items added to the target object can show more realistic fabric texture and natural lighting effects, accurately reproduce the details such as the logo, pattern, text and stripes of the target item, and improve the robustness of the generated target image, thereby achieving the technical effect of improving the fidelity and detail restoration of the virtual try-on image, and then solving the technical problem of low fidelity and detail restoration of the virtual try-on image in the scene of multiple clothing.

[0206] Those skilled in the art will appreciate that the structure shown in the above figure is for illustration only, and the computer terminal A may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. The above figure does not limit the structure of the above electronic devices. For example, the computer terminal A may also include more or fewer components (such as a network interface, a display device, etc.) than those shown in the above figure, or have a different configuration than that shown in the above figure.

[0207] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0208] The embodiment of the present application also provides a computer-readable storage medium. In the above embodiment of the present application, in this embodiment, the above computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.

[0209] In the above embodiment of the present application, in this embodiment, the above computer-readable storage medium may be located in any one of the computer terminals in the computer terminal cluster, or in any one of the mobile terminals in the mobile terminal group.

[0210] In the above-mentioned embodiment of the present application, in this embodiment, a computer-readable storage medium is configured to store a program code for executing the following steps: in response to an input instruction acting on an operation interface, a first image of a target object and a second image of a plurality of target items are displayed on the operation interface, wherein different target items are used to be added at different positions of the target object; in response to an image generation instruction acting on the operation interface, a target image is displayed on the operation interface, wherein a plurality of target items are added to the target object in the target image, and the target image is an image generated by guiding an image generation model based on item features of the plurality of target items and the first image, and the item features of the plurality of target items are features obtained by encoding the second images of the plurality of target items using an item coding model.

[0211] In the above embodiment of the present application, the computer-readable storage medium is also configured to store program codes for executing the following steps: the item coding model includes: multiple feature extraction networks, wherein the multiple feature extraction networks correspond to multiple target items one by one, the multiple feature extraction networks are executed in parallel, and the multiple feature extraction networks share some weights. The method also includes: inputting the second images of the multiple target items into the corresponding feature extraction networks respectively, and obtaining the item features output by the corresponding feature extraction networks to obtain the item features of the multiple target items.

[0212] In the above embodiment of the present application, the computer-readable storage medium is also configured to store program codes for executing the following steps: each feature extraction network includes: at least one feature extraction module, each feature extraction module includes a first self-attention layer and a first cross-attention layer, wherein multiple feature extraction networks share the weight of the first cross-attention layer; the second images of multiple target items are respectively input into the corresponding feature extraction networks, and the item features output by the corresponding feature extraction networks are obtained, including: using the first self-attention layer to perform self-attention processing on the input features of the feature extraction module to obtain a first self-attention feature, wherein, when the feature extraction module is the first feature extraction module in at least one feature extraction module, the input feature of the feature extraction module is the second image of the corresponding target item, and when the feature extraction module is not the first feature extraction module, the input feature of the feature extraction module is the output feature of the previous feature extraction module; using the first cross-attention layer to perform cross-attention processing on the first self-attention feature and the first preset feature to obtain the output feature of the feature extraction module; determining the output feature of the last feature extraction module in at least one feature extraction module as the item feature output by the corresponding feature extraction network.

[0213] In the above embodiment of the present application, the computer-readable storage medium is also configured to store program codes for executing the following steps: the method also includes: splicing the object features of multiple target objects to obtain splicing features; extracting features of the first image to obtain image features of the first image; inputting the splicing features and the image features into the image generation model, and obtaining the target image output by the image generation model.

[0214] In the above embodiments of the present application, the computer-readable storage medium is also configured to store program codes for executing the following steps: splicing the item features of multiple target items to obtain splicing features, including: superimposing the item features and corresponding position features of multiple target items to obtain multiple superimposed features, wherein different target items have different corresponding position features; splicing multiple superimposed features to obtain a splicing feature.

[0215] In the above-mentioned embodiment of the present application, the computer-readable storage medium is also configured to store program codes for executing the following steps: extracting features from the first image to obtain image features of the first image, including: extracting skeleton information of the target object in the first image to obtain a skeleton image of the target object, and extracting features from the skeleton image to obtain skeleton features of the target object; and extracting features from a mask image of the first image to obtain mask features of the first image, wherein the mask image is an image determined in response to an input instruction, or an image obtained by processing the first image based on the skeleton image; processing the first image to obtain a third image based on the position of the original object in the first image, and extracting features from the third image to obtain background features of the first image, wherein the original object is not included in the third image; and splicing the skeleton features, mask features and background features to obtain image features.

[0216] In the above embodiment of the present application, the computer-readable storage medium is also configured to store program codes for executing the following steps: the image generation model includes: at least one image generation module, each image generation module includes a second self-attention layer and a second cross-attention layer; the splicing features and the image features are input into the image generation model, and the target image output by the image generation model is obtained, including: using the second self-attention layer to perform self-attention processing on the input features and the splicing features of the image generation module to obtain the second self-attention features, wherein, when the image generation module is the first image generation module in at least one image generation module, the input features of the image generation module are the image features, and when the image generation module is not the first image generation module, the input features of the image generation module are the output features of the previous image generation module; using the second cross-attention layer to perform cross-attention processing on the second self-attention features and the second preset features to obtain the output features of the image generation module; based on the output features of the last image generation module in at least one image generation module, the target image is generated.

[0217] In the above embodiment of the present application, the computer-readable storage medium is also configured to store program codes for executing the following steps: using the second self-attention layer to perform self-attention processing on the input features and splicing features of the image generation module to obtain a second self-attention feature, including: determining the query vector corresponding to the second self-attention layer based on the input features of the image generation module; splicing the input features and splicing features of the image generation module to obtain a key vector and a value vector corresponding to the second self-attention layer; using the query vector, the key vector and the value vector to obtain the second self-attention feature.

[0218] In the above embodiment of the present application, the computer-readable storage medium is also configured to store program codes for executing the following steps: the method also includes: obtaining a target text, wherein the target text is used to describe a target image; splicing the second images of multiple target objects to obtain a spliced ​​image; and performing feature extraction on the target text and the target image using a multimodal processing model to obtain a second preset feature, wherein the second preset feature is used to characterize the item information of the multiple target objects and the method of adding the multiple target objects to the target object.

[0219] In the above embodiments of the present application, the computer-readable storage medium is also configured to store program codes for executing the following steps: the initial network parameters of the image generation model are obtained by fusing the network parameters of multiple preset models, and the network structures of different preset models are the same but the network parameters are different.

[0220] In the above embodiments of the present application, the computer-readable storage medium is also configured to store program codes for executing the following steps: the method also includes: determining fusion weights corresponding to multiple preset models; fusing network parameters of multiple preset models based on the fusion weights corresponding to multiple preset models to obtain initial network parameters.

[0221] The embodiment of the present application also provides a computer program product. In the above embodiment of the present application, in this embodiment, the above computer program product may include a computer program, and the above computer program implements the method provided in the above embodiment when executed by a processor.

[0222] The embodiments of the present application also provide a computer program product. In the above embodiments of the present application, the above computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program, and when the computer program is executed by a processor, the method provided in the above embodiments is implemented.

[0223] The embodiment of the present application also provides a computer program. In the above embodiment of the present application, in this embodiment, when the above computer program is executed by a processor, the method provided in the above embodiment is implemented.

[0224] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0225] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0226] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0227] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0228] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0229] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image generation method, characterized in that: include: In response to an input instruction acting on an operation interface, a first image of a target object and second images of a plurality of target items are displayed on the operation interface, wherein different target items are used to be added at different positions of the target object; In response to an image generation instruction applied to the operation interface, a target image is displayed on the operation interface, wherein the target object in the target image has the multiple target items added thereto, and the target image is an image generated by guiding an image generation model based on item features of the multiple target items and the first image, and the item features of the multiple target items are features obtained by encoding the second images of the multiple target items using an item coding model.

2. The method according to claim 1, characterized in that The item coding model includes: a plurality of feature extraction networks, wherein the plurality of feature extraction networks correspond to the plurality of target items one by one, the plurality of feature extraction networks are executed in parallel, and the plurality of feature extraction networks share some weights; The method further comprises: The second images of the multiple target objects are respectively input into the corresponding feature extraction network, and the object features output by the corresponding feature extraction network are obtained to obtain the object features of the multiple target objects.

3. The method according to claim 2, characterized in that Each feature extraction network comprises: at least one feature extraction module, each feature extraction module comprises a first self-attention layer and a first cross-attention layer, wherein the plurality of feature extraction networks share the weight of the first cross-attention layer; Inputting the second images of the plurality of target objects into corresponding feature extraction networks respectively, and obtaining object features output by the corresponding feature extraction networks, including: Using the first self-attention layer to perform self-attention processing on the input feature of the feature extraction module to obtain a first self-attention feature, wherein, when the feature extraction module is the first feature extraction module of the at least one feature extraction module, the input feature of the feature extraction module is the second image of the corresponding target object, and when the feature extraction module is not the first feature extraction module, the input feature of the feature extraction module is the output feature of the previous feature extraction module; Using the first cross-attention layer to perform cross-attention processing on the first self-attention feature and the first preset feature to obtain an output feature of the feature extraction module; Determine the output feature of the last feature extraction module in the at least one feature extraction module as the item feature output by the corresponding feature extraction network.

4. The method according to claim 1, characterized in that The method further comprises: splicing the item features of the multiple target items to obtain a spliced ​​feature; Performing feature extraction on the first image to obtain image features of the first image; The splicing feature and the image feature are input into the image generation model, and the target image output by the image generation model is obtained.

5. The method according to claim 4, characterized in that The item features of the multiple target items are spliced ​​to obtain spliced ​​features, including: Superimposing the item features and corresponding position features of the multiple target items to obtain multiple superimposed features, wherein different target items have different corresponding position features; The multiple superimposed features are spliced ​​together to obtain the spliced ​​feature.

6. The method according to claim 4, characterized in that The extracting features of the first image to obtain image features of the first image includes: Extracting skeleton information of the target object in the first image to obtain a skeleton image of the target object, and performing feature extraction on the skeleton image to obtain skeleton features of the target object; and extracting features from a mask image of the first image to obtain mask features of the first image, wherein the mask image is an image determined in response to the input instruction, or an image obtained by processing the first image based on the skeleton image; Based on the position of the original object in the first image, the first image is processed to obtain a third image, and features are extracted from the third image to obtain background features of the first image, wherein the third image does not contain the original object; The skeleton feature, the mask feature and the background feature are concatenated to obtain the image feature.

7. The method according to claim 4, characterized in that The image generation model comprises: at least one image generation module, each image generation module comprising a second self-attention layer and a second cross-attention layer; Inputting the splicing feature and the image feature into the image generation model, and obtaining the target image output by the image generation model, comprises: Using the second self-attention layer to perform self-attention processing on the input features of the image generation module and the splicing features to obtain a second self-attention feature, wherein, when the image generation module is the first image generation module in the at least one image generation module, the input feature of the image generation module is the image feature, and when the image generation module is not the first image generation module, the input feature of the image generation module is the output feature of the previous image generation module; Using the second cross-attention layer to perform cross-attention processing on the second self-attention feature and the second preset feature to obtain an output feature of the image generation module; The target image is generated based on the output features of the last image generation module in the at least one image generation module.

8. The method according to claim 7, characterized in that The second self-attention layer is used to perform self-attention processing on the input features of the image generation module and the splicing features to obtain a second self-attention feature, including: Determine a query vector corresponding to the second self-attention layer based on the input features of the image generation module; Splicing the input features of the image generation module and the splicing features to obtain a key vector and a value vector corresponding to the second self-attention layer; The second self-attention feature is obtained using the query vector, the key vector, and the value vector.

9. The method according to claim 7, characterized in that: The method further comprises: Acquire a target text, wherein the target text is used to describe the target image; Stitching the second images of the plurality of target objects to obtain a stitched image; The target text and the spliced ​​image are subjected to feature extraction using a multimodal processing model to obtain the second preset feature, wherein the second preset feature is used to characterize the item information of the multiple target items and the manner in which the multiple target items are added to the target object.

10. The method according to any one of claims 1 to 9, characterized in that The initial network parameters of the image generation model are obtained by fusing the network parameters of multiple preset models. Different preset models have the same network structure but different network parameters.

11. The method according to claim 10, characterized in that The method further comprises: Determining fusion weights corresponding to the multiple preset models; The network parameters of the multiple preset models are fused based on the fusion weights corresponding to the multiple preset models to obtain the initial network parameters.

12. An image generation method, characterized in that: include: Acquire a first image of a target object and second images of a plurality of target objects, wherein different target objects are used to be added at different positions of the target object; Encoding the second images of the plurality of target objects using an object coding model to obtain object features of the plurality of target objects; Based on the item features of the multiple target items and the first image, an image generation model is guided to generate a target image, wherein the multiple target items are added to the target object in the target image.

13. An image generation method, characterized in that: include: In response to an input instruction acting on the operation interface, displaying a first image of a biological object and second images of a plurality of clothing items on the operation interface, wherein different clothing items are used to be worn on different parts of the biological object; In response to an image generation instruction applied to the operation interface, a target image is displayed on the operation interface, wherein the biological object in the target image is wearing the multiple pieces of clothing, and the target image is an image generated based on clothing features of the multiple pieces of clothing and the first image-guided image generation model, and the clothing features of the multiple pieces of clothing are features obtained by encoding a second image of the multiple pieces of clothing using a clothing coding model.

14. An image generation method, characterized in that: include: Acquire a first image of a target object and second images of a plurality of target objects by calling a first interface, wherein the first interface includes a first parameter, a parameter value of the first parameter includes the first image and the second images of the plurality of target objects, and different target objects are used to be added at different positions of the target object; Encoding the second images of the plurality of target objects using an object coding model to obtain object features of the plurality of target objects; Based on the item features of the multiple target items and the first image, guiding the image generation model to generate a target image, wherein the multiple target items are added to the target object in the target image; The target image is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter includes the target image.

15. A computer terminal, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 14 when running.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 14.

17. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images