Image generation method, target text image generation model generation method and device

Abstract

The present application discloses an image generation method, a generation method and a device of a target text image generation model. The present application relates to the field of artificial intelligence, and the method comprises: receiving a foreground image containing a target object and text information corresponding to the target object; determining the description information and layout information of the target scene corresponding to the target object according to the foreground image and text information through a target multimodal visual language model; generating a scene graph of the target object according to the foreground image, description information and layout information through a target text image generation model, wherein the target text image generation model is obtained by training an initial text image generation model according to a training sample set. The present application solves the technical problem in the related art that the scene graph of an object is generated according to the manually input text description information, and the accuracy of the text description information is low, resulting in poor visual effects of the generated scene graph.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to an image generation method, a method and a device for generating a target text image generative model. Background Art

[0002] To create exquisite creative advertising images, such as taking pictures of sofas placed in a cozy room, requires certain photography and design skills, and requires merchants to invest a lot of manpower, material and financial resources. To help merchants quickly and cost-effectively create high-quality creative images and improve advertising effectiveness, artificial intelligence models can be used to generate scene maps for products.

[0003] At present, the commonly used method in related technologies is that users upload the product foreground image, manually select or input the text description (prompt) of the scene graph, and use the text image generation model to generate the background. The model can output a scene graph with the background completed according to the prompt. The following problems exist in this process, which often lead to poor visual effects of the generated scene graph: it is difficult to have a suitable text description, that is, the accuracy of the text description is low, resulting in the generated scene not fitting the product and the foreground and background scenes not matching; the size and position of the product foreground are fixed, and it is easy to have the position of the generated image elements disordered, and the deformity rate is high; the existing model is insufficient, and the number and size of the generated objects are likely to be inconsistent with the real physical laws.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide an image generation method, a generation method and a device for a target text-based image generation model, so as to at least solve the technical problem in the related art of generating a scene graph of an object based on manually input text description information, which has low accuracy of the text description information and leads to poor visual effect of the generated scene graph.

[0006] According to one aspect of an embodiment of the present application, an image generation method is provided, including: receiving a foreground image containing a target object and text information corresponding to the target object; determining, by a target multimodal visual language model, description information and layout information of a target scene corresponding to the target object based on the foreground image and the text information; generating, by a target text-image generative model, a scene graph of the target object based on the foreground image, the description information and the layout information, wherein the target text-image generative model is obtained by training an initial text-image generative model based on a training sample set.

[0007] Furthermore, the target multimodal visual language model includes a first model and a second model. Determining the description information and layout information of the target scene corresponding to the target object through the target multimodal visual language model based on the foreground image and text information includes: determining the description information through the first model based on the foreground image and text information, wherein the first model is obtained by training the scene description output capability of the initial multimodal visual language model based on the first training sample set; determining the layout information through the second model based on the foreground image and the description information, wherein the second model is obtained by training the layout information generation capability of the initial multimodal visual language model based on the second training sample set.

[0008] Furthermore, determining the description information based on the foreground image and the text information through the first model includes: processing the foreground image and the text information respectively through the first model to obtain an image feature representation corresponding to the foreground image and a text feature representation corresponding to the text information; processing the image feature representation and the text feature representation based on the self-attention mechanism to obtain a fused feature representation, and predicting the description information based on the fused feature representation.

[0009] Furthermore, determining the layout information through the second model based on the foreground image and the description information includes: extracting the target object from the foreground image through the second model, and extracting multiple background elements from the description information; predicting the size and position based on the target object and the multiple background elements to obtain the size and position information of the target object, as well as the size and position information of the background elements among the multiple background elements; and composing the layout information based on the size and position information of the target object, as well as the size and position information of the background elements among the multiple background elements.

[0010] Furthermore, generating a scene graph of a target object according to a foreground image, description information and layout information through a target text-based image generative model includes: extracting a target object from a foreground image through a target text-based image generative model, and adjusting the size and position of the target object according to the size and position information of the target object in the layout information to obtain an initial scene graph; controlling the foreground pixels of the target object in the initial scene graph to be fixed according to the description information and the layout information, generating a background area corresponding to the initial scene graph, and obtaining a scene graph of the target object.

[0011] Furthermore, generating a background area corresponding to the initial scene graph to obtain a scene graph of the target object includes: processing the background elements in the layout information through a target text graph generative model to obtain a marker embedding vector, and processing the position information of the background elements in the multiple background elements in the layout information to obtain a position embedding vector; concatenating the marker embedding vector and the position embedding vector to obtain a target embedding vector; inputting the target embedding vector into a position information control module of the target text graph generative model, so that the position information control module generates a background area corresponding to the initial scene graph according to the target embedding vector to obtain the scene graph of the target object.

[0012] According to another aspect of an embodiment of the present application, a method for generating a target Wensheng graph generative model is also provided, including: obtaining a training sample set, wherein the training sample set includes: multiple training sample pairs, the training sample pairs include sample input and sample output, the sample input consists of a foreground image containing a sample object, a sample scene description and sample layout information, and the sample output is a sample scene graph corresponding to the sample input; after the self-attention mechanism of the image generation network of the initial Wensheng graph generative model, adding a position information control module to obtain a Wensheng graph generative model to be trained, wherein the position information control module is used to control the image generation process of the image generation network according to the sample layout information; training the Wensheng graph generative model to be trained according to the training sample set to obtain a trained Wensheng graph generative model; training the trained Wensheng graph generative model according to the target control network to obtain a target Wensheng graph generative model, wherein the target control network is used to control the object shape of the sample object.

[0013] According to another aspect of an embodiment of the present application, an image generation method is also provided, including: obtaining a foreground image containing a target object and text information corresponding to the target object uploaded by a client; determining, in a cloud server, description information and layout information of a target scene corresponding to the target object based on the foreground image and text information through a target multimodal visual language model; generating a scene graph of the target object based on the foreground image, description information and layout information through a target text graph generative model, wherein the target text graph generative model is obtained by training an initial text graph generative model based on a training sample set; and feeding back the scene graph of the target object to the client.

[0014] According to another aspect of an embodiment of the present application, an image generation device is also provided, including: a receiving unit, configured to receive a foreground image containing a target object and text information corresponding to the target object; a determining unit, configured to determine description information and layout information of a target scene corresponding to the target object based on the foreground image and the text information through a target multimodal visual language model; and a processing unit, configured to generate a scene graph of the target object based on the foreground image, the description information and the layout information through a target text graph generative model, wherein the target text graph generative model is obtained by training an initial text graph generative model based on a training sample set.

[0015] Furthermore, the determination unit includes: a first determination subunit, used to determine the description information based on the foreground image and text information through a first model, wherein the first model is obtained by training the scene description output capability of the initial multimodal visual language model based on the first training sample set; a second determination subunit, used to determine the layout information based on the foreground image and the description information through a second model, wherein the second model is obtained by training the layout information generation capability of the initial multimodal visual language model based on the second training sample set.

[0016] Furthermore, the first determination subunit includes: a first processing module, used to process the foreground image and text information respectively through the first model to obtain the image feature representation corresponding to the foreground image and the text feature representation corresponding to the text information; a second processing module, used to process the image feature representation and the text feature representation based on the self-attention mechanism to obtain a fused feature representation, and predict the description information based on the fused feature representation.

[0017] Furthermore, the second determination subunit includes: a third processing module, used to extract the target object from the foreground image through the second model, and extract multiple background elements from the description information; a fourth processing module, used to predict the size and position based on the target object and multiple background elements, and obtain the size and position information of the target object, as well as the size and position information of the background elements among the multiple background elements; a fifth processing module, used to compose layout information based on the size and position information of the target object, as well as the size and position information of the background elements among the multiple background elements.

[0018] Furthermore, the processing unit includes: a first processing sub-unit, used to extract the target object from the foreground image through the target text image generation model, and adjust the size and position of the target object according to the size and position information of the target object in the layout information to obtain an initial scene graph; a second processing sub-unit, used to control the foreground pixels of the target object in the initial scene graph to be fixed according to the description information and the layout information, generate a background area corresponding to the initial scene graph, and obtain a scene graph of the target object.

[0019] Furthermore, the second processing sub-unit includes: a sixth processing module, which is used to process the background elements in the layout information through the target text image generation model to obtain a mark embedding vector, and process the position information of the background elements in the multiple background elements in the layout information to obtain a position embedding vector; a seventh processing module, which is used to splice the mark embedding vector and the position embedding vector to obtain a target embedding vector; and an eighth processing module, which is used to input the target embedding vector into the position information control module of the target text image generation model, so that the position information control module generates a background area corresponding to the initial scene graph according to the target embedding vector to obtain the scene graph of the target object.

[0020] According to another aspect of an embodiment of the present application, a device for generating a target Wensheng graph generative model is also provided, including: an acquisition unit, used to acquire a training sample set, wherein the training sample set includes: a plurality of training sample pairs, the training sample pairs include a sample input and a sample output, the sample input consists of a foreground image containing a sample object, a sample scene description and sample layout information, and the sample output is a sample scene graph corresponding to the sample input; a construction unit, used to add a position information control module after the self-attention mechanism of the image generation network of the initial Wensheng graph generative model to obtain a Wensheng graph generative model to be trained, wherein the position information control module is used to control the image generation process of the image generation network according to the sample layout information; a first training unit, used to train the Wensheng graph generative model to be trained according to the training sample set to obtain a trained Wensheng graph generative model; a second training unit, used to train the trained Wensheng graph generative model according to the target control network to obtain a target Wensheng graph generative model, wherein the target control network is used to control the object shape of the sample object.

[0021] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein any one of the above-mentioned image generation methods is executed when the program is run.

[0022] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a program, wherein when the program is running, the device where the storage medium is located is controlled to execute any one of the above-mentioned image generation methods.

[0023] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program or instructions, and when the computer program or instructions are executed by a processor, any one of the above-mentioned image generation methods is implemented.

[0024] In an embodiment of the present application, a foreground image containing a target object and text information corresponding to the target object are received; description information and layout information of a target scene corresponding to the target object are determined based on the foreground image and text information by a target multimodal visual language model; a scene graph of the target object is generated based on the foreground image, description information and layout information by a target text graph generative model, wherein the target text graph generative model is obtained by training an initial text graph generative model based on a training sample set, and the target multimodal visual language model can generate a suitable scene graph description based on the object according to the knowledge learned by the target multimodal visual language model, thereby improving the fit between the scene and the object, improving the fit between the foreground and background scenes of the scene graph, and can also predict the object and the scene. The layout information of elements can effectively avoid problems such as unreasonable object size, inappropriate object placement, and unreasonable generation of scene elements; using the target text-generated image generation model to generate a scene graph according to the foreground image, description information and layout information, the object features can be effectively maintained, and the placement relationship between the target object and the surrounding objects is more in line with the actual objective laws, which improves the naturalness and availability of the image, and achieves the purpose of improving the quality of the raw image of the text-generated image generation model, thereby achieving the technical effect of improving the accuracy of the text description information and improving the visual effect of the generated scene graph, and further solves the technical problem that the scene graph of the object is generated based on the manually input text description information in the related technology, and the accuracy of the text description information is low, resulting in poor visual effect of the generated scene graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0026] Figure 1 is a schematic diagram of a computer terminal provided according to Embodiment 1 of the present application;

[0027] Figure 2 is a flowchart of an image generation method provided according to Embodiment 1 of the present application;

[0028] Figure 3 It is a schematic diagram of a process for generating an optional scene graph according to the first embodiment of the present application;

[0029] Figure 4 It is a schematic diagram of a flow chart of generating an optional scene graph description according to the first embodiment of the present application;

[0030] Figure 5 is a flow chart of a method for generating a target text graph generative model according to the second embodiment of the present application;

[0031] Figure 6is a flowchart of an image generation method provided according to Embodiment 3 of the present application;

[0032] Figure 7 is a schematic diagram of an image generating device provided according to Embodiment 4 of the present application;

[0033] Figure 8 It is a schematic diagram of a device for generating a target text graph generative model according to Embodiment 5 of the present application;

[0034] Fig. 9 This is a structural block diagram of an electronic device provided according to Embodiment 6 of the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0038] Example 1

[0039] According to an embodiment of the present application, an image generating method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image generation method. Figure 1 As shown, the computer terminal (or mobile device) 10 may include a processor set 102 (the processor set 102 may include but is not limited to a processing device such as a microprocessor (Microcontroller Unit, referred to as MCU) or a programmable logic device (Field Programmable Gate Array, referred to as FPGA), and the processor set 102 may include a processor set, Figure 1 102a, 102b, ..., 102n are used to illustrate), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0041] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuits may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device).

[0042] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the image generation method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned image generation method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0043] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0044] The display may be a touch screen type liquid crystal display, which may enable a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0045] To create exquisite creative advertising images, such as taking pictures of sofas placed in a cozy room, requires certain photography and design skills, and requires merchants to invest a lot of manpower, material and financial resources. To help merchants quickly and cost-effectively create high-quality creative images and improve advertising effectiveness, artificial intelligence models can be used to generate scene maps for products.

[0046] At present, the commonly used method in related technologies is that users upload the product foreground image, manually select or input the text description (prompt) of the scene graph, and use the text image generation model to generate the background. The model can output a scene graph with the background completed according to the prompt. The following problems exist in this process, which often lead to poor visual effects of the generated scene graph: it is difficult to have a suitable text description, that is, the accuracy of the text description is low, resulting in the generated scene not fitting the product and the foreground and background scenes not matching; the size and position of the product foreground are fixed, and it is easy to have the position of the generated image elements disordered, and the deformity rate is high; the existing model is insufficient, and the number and size of the generated objects are likely to be inconsistent with the real physical laws.

[0047] In the above technical background, the present application provides Figure 2 The image generation method shown. Figure 2 : is a flow chart of an image generation method provided according to Embodiment 1 of the present application. The method comprises:

[0048] Step S201: receiving a foreground image containing a target object and text information corresponding to the target object.

[0049] Optionally, the image generation system is used as the executor. For example, the user wants to generate a scene graph for a product, that is, to complete the background based on the product foreground to obtain a scene graph containing the product foreground and the scene background. The image generation system can receive the foreground image containing the target object and the text information corresponding to the target object input by the user. For example, the target object can be a product such as a sofa, and the corresponding text information can be the product title, category, brand, style, material, selling point information, etc.

[0050] Step S202, determining description information and layout information of a target scene corresponding to a target object based on the foreground image and text information through a target multimodal visual language model.

[0051] Optionally, the multimodal visual language model is an artificial intelligence model that can simultaneously process and understand both visual (image) and language (text) modal information. In an optional embodiment, the target multimodal visual language model includes a first model and a second model, the first model is obtained by training the scene description output capability of the initial multimodal visual language model based on the first training sample set, and the second model is obtained by training the layout information generation capability of the initial multimodal visual language model based on the second training sample set.

[0052] For example, an initial multimodal visual language model is trained based on a first training sample set constructed from sample product foregrounds and sample product titles and other text information, and a multimodal visual language model (i.e., a first model) is obtained that can predict a reasonable scene graph description (prompt) based on the product foreground and text information. An initial multimodal visual language model is trained based on a second training sample set constructed from sample product foregrounds and sample scene graph prompts, and a multimodal visual language model (i.e., a second model) is obtained that can predict the layout information of products and background elements based on product foregrounds and scene graph prompts.

[0053] Therefore, the image generation system can determine the description information (prompt) and layout information of the target scene corresponding to the target object based on the foreground image and text information through the target multimodal visual language model.

[0054] Step S203, generating a scene graph of the target object according to the foreground image, the description information and the layout information by using a target text graph generative model, wherein the target text graph generative model is obtained by training an initial text graph generative model according to a training sample set.

[0055] Optionally, the initial text image generation model adopts the text image generation model Stable Diffusion (SD for short). In order to keep the size and position of the object consistent with the prompt and ensure that the foreground feature of the object is maintained, the SD model is modified, that is, after the self-attention mechanism of each layer of the U-net network of the SD model, a position information control module is added to control the image generation process according to the layout information, and guide the text image model to generate specified elements at a given position. Fine-tune the modified model according to the training sample set, and add the control network (inpainting controlNet) for joint training in the later stage of training, so as to obtain a text image generation model (i.e., a target text image generation model) that accepts joint control of foreground, prompt and layout at the same time. Therefore, the image generation system can input the product foreground, scene image prompt and layout information into the target text image generation model, and output a scene image with reasonable spatial semantics, high quality and high aesthetics through the target text image generation model.

[0056] In this scheme, the target multimodal visual language model can generate a suitable scene graph description according to the object based on the knowledge it has learned, thereby improving the fit between the scene and the object, and improving the fit between the foreground and background scenes of the object in the scene graph. It can also predict the layout information of the object and the scene elements, and effectively avoid problems such as unreasonable object size, inappropriate object placement, and unreasonable generation of scene elements; the target text-generated graph generation model is used to generate a scene graph based on the foreground image, description information, and layout information, so that the object features can be effectively maintained, and the placement relationship between the target object and the surrounding objects is more in line with the actual objective laws, which improves the naturalness and availability of the image, and achieves the purpose of improving the quality of the raw image of the text-generated graph generation model, thereby achieving the technical effect of improving the accuracy of the text description information and improving the visual effect of the generated scene graph, and then solving the technical problem that the scene graph of the object is generated based on the manually input text description information in the related technology, and the accuracy of the text description information is low, resulting in poor visual effect of the generated scene graph.

[0057] In an optional embodiment, it is possible to use Figure 3 The schematic diagram shown in FIG. 1 realizes scene graph generation. Figure 3 is a schematic diagram of a flow chart of optional scene graph generation according to the first embodiment of the present application, such as Figure 3As shown in the figure, in order to solve the common problems in the image generation process, such as the mismatch between product scenes, improper placement of products, and the generated images not conforming to physical laws, the problems are decomposed and a multi-step image generation process is constructed. First, a multimodal visual language model is used to generate a reasonable scene image description prompt based on the product foreground image and text information such as the title. Then, the multimodal visual language model is used to generate a layout of the product foreground and background elements based on the prompt and the product foreground. Then, the size and position of the product foreground are adjusted according to the layout. The background is filled in with a text-generated image generation model that can accept the joint control of the foreground, prompt and layout to generate a complete scene image.

[0058] In order to generate a reasonable scene graph description prompt and the layout of product foreground and background elements, in the image generation method provided in Example 1 of the present application, the target multimodal visual language model includes a first model and a second model. Determining the description information and layout information of the target scene corresponding to the target object based on the foreground image and text information through the target multimodal visual language model includes: determining the description information based on the foreground image and text information through the first model, wherein the first model is obtained by training the scene description output capability of the initial multimodal visual language model based on the first training sample set; determining the layout information based on the foreground image and the description information through the second model, wherein the second model is obtained by training the layout information generation capability of the initial multimodal visual language model based on the second training sample set.

[0059] Optionally, the description information can be determined by the first model based on the foreground image and text information. For example, the product foreground image and text information such as the title are input into the first model, and a reasonable scene image description prompt (i.e., description information) is predicted by the first model.

[0060] Optionally, the second model can determine the layout information based on the foreground image and description information. For example, the product foreground image and the scene image description prompt are input into the second model, and the layout information of the product and background elements is predicted by the second model.

[0061] It should be noted that the target multimodal visual language model can generate appropriate scene graph descriptions based on objects based on the knowledge it has learned, thereby improving the fit between the scene and the object, and the fit between the foreground and background scenes of the scene graph. It can also predict the layout information of objects and scene elements, effectively avoiding problems such as unreasonable object size, inappropriate object placement, and unreasonable generation of scene elements.

[0062] In order to obtain a reasonable scene graph description prompt, in the image generation method provided in Example 1 of the present application, the description information is determined by the first model based on the foreground image and text information, including: the foreground image and text information are processed separately by the first model to obtain the image feature representation corresponding to the foreground image and the text feature representation corresponding to the text information; the image feature representation and the text feature representation are processed based on the self-attention mechanism to obtain a fused feature representation, and the description information is predicted based on the fused feature representation.

[0063] Optionally, the foreground image and text information are processed separately by the first model to obtain image feature representation corresponding to the foreground image and text feature representation corresponding to the text information. Then, the image feature representation and the text feature representation are processed based on the self-attention mechanism to obtain a fused feature representation, and the description information is predicted based on the fused feature representation.

[0064] For example, the foreground image is processed through the multimodal visual language model to obtain the token of the image feature (i.e., the image feature representation), and the text information is processed to obtain the token of the text (i.e., the text feature representation). Then, the self-attention mechanism is operated in the model space. The image feature token and the text token can perceive each other's information, and the image feature token fused with the text and the text token fused with the image are generated. Then, the fused feature representation is generated based on the two fused tokens. Then, the fused feature representation is analyzed according to the distribution characteristics of the previously learned training data, and the scene graph description prompt can be predicted.

[0065] In an optional embodiment, it is possible to use Figure 4 The schematic diagram shown in FIG. 4 realizes the scene graph description generation. Figure 4 is a schematic diagram of a flow chart of generating an optional scene graph description according to the first embodiment of the present application, such as Figure 4 As shown in the figure, the product foreground image, product title and other text information are input into the multimodal visual language model to generate a better scene graph description prompt, so that the knowledge accumulated by the multimodal visual language model can be used to achieve high-quality, high-degree-of-freedom and high-diversity image generation.

[0066] It should be noted that the multimodal visual language model can generate a suitable scene graph description based on the object based on the knowledge it has learned, which improves the fit between the scene and the object, and improves the fit between the foreground and background scenes of the scene graph.

[0067] In order to obtain the layout of the foreground and background elements of the product, in the image generation method provided in the first embodiment of the present application, the layout information is determined by the second model based on the foreground image and the description information, including: extracting the target object from the foreground image through the second model, and extracting multiple background elements from the description information; predicting the size and position based on the target object and the multiple background elements to obtain the size and position information of the target object, as well as the size and position information of the background elements among the multiple background elements; and composing the layout information based on the size and position information of the target object, as well as the size and position information of the background elements among the multiple background elements.

[0068] Optionally, a target object is extracted from the foreground image through a second model, and multiple background elements are extracted from the description information. Then, size and position predictions are performed based on the target object and the multiple background elements to obtain size and position information (such as position coordinates) of the target object and size and position information of the background elements among the multiple background elements. Then, layout information is composed based on the size and position information of the target object and the size and position information of the background elements among the multiple background elements.

[0069] For example, the layout can be further generated based on the product foreground and scene graph description prompt through the multimodal visual language model. For example, the product is a sofa, and the scene graph description prompt is: the sofa is placed on the wooden floor, and there is a carpet next to the wooden floor. The multimodal visual language model can extract background elements (such as background element words such as wooden floor and carpet) from the scene graph description prompt. Since the multimodal visual language model has seen many pictures and has certain spatial knowledge, it can predict the layout information of the product and background elements based on the product foreground, scene graph description prompt and background elements.

[0070] It should be noted that the multimodal visual language model can predict the layout information of objects and scene elements based on the knowledge it has learned, effectively avoiding problems such as unreasonable object size, inappropriate object placement, and unreasonable generation of scene elements.

[0071] In order to obtain a scene graph of a commodity, in the image generation method provided in the first embodiment of the present application, a scene graph of a target object is generated according to a foreground image, description information and layout information through a target text image generative model, including: extracting a target object from a foreground image through a target text image generative model, and adjusting the size and position of the target object according to the size and position information of the target object in the layout information to obtain an initial scene graph; controlling the foreground pixels of the target object in the initial scene graph to be fixed according to the description information and layout information, generating a background area corresponding to the initial scene graph, and obtaining a scene graph of the target object.

[0072] Optionally, the target object is extracted from the foreground image through a target scene graph generation model, and the size and position of the target object are adjusted according to the size and position information of the target object in the layout information, so as to obtain an initial scene graph. Then, the foreground pixels of the target object in the initial scene graph are controlled to be fixed according to the description information and the layout information, and the background area corresponding to the initial scene graph is generated, so as to obtain a scene graph of the target object.

[0073] For example, the product foreground, scene graph description prompt and layout information are input into the target text image generation model, and the target text image generation model is used to adjust the size and position information of the product in the layout information to obtain the initial scene graph. Then, the foreground pixels of the product in the initial scene graph are controlled to be fixed according to the scene graph description prompt and layout information, and the background area is generated to obtain the scene graph of the product.

[0074] It should be noted that using the target text-based image generation model to generate a scene graph based on the foreground image, description information, and layout information can effectively maintain object features, and the placement relationship between the target object and surrounding objects is more in line with actual objective laws, thereby improving the naturalness and usability of the image.

[0075] In order to obtain a scene graph of a commodity, in the image generation method provided in the first embodiment of the present application, a background area corresponding to an initial scene graph is generated to obtain a scene graph of a target object, including: processing background elements in layout information through a target text graph generation model to obtain a marker embedding vector, and processing position information of background elements in multiple background elements in the layout information to obtain a position embedding vector; concatenating the marker embedding vector and the position embedding vector to obtain a target embedding vector; and inputting the target embedding vector into a position information control module of a target text graph generation model so that the position information control module generates a background area corresponding to the initial scene graph according to the target embedding vector to obtain a scene graph of the target object.

[0076] Optionally, by processing the background elements in the layout information through the target text-based graph generative model, a mark embedding vector can be obtained, and by processing the position information of the background elements in the multiple background elements in the layout information, a position embedding vector can be obtained, and then the mark embedding vector and the position embedding vector are concatenated to obtain a target embedding vector, and then the target embedding vector is input into the position information control module of the target text-based graph generative model, so that the position information control module generates a background area corresponding to the initial scene graph according to the target embedding vector to obtain a scene graph of the target object.

[0077] For example, the background element words in the layout information are processed by the text encoder of the target text-based graph generation model to obtain the token embedding vectors of the background element words. The position coordinates of the background element words are transformed by the target text-based graph generation model to obtain the position embedding vectors of the background element words. The token embedding and position embedding of the background element words are then concatenated as a control signal (i.e., the target embedding vector). The control signal is input into the position information control module to guide the text-based graph model to generate a specified element at a given position, i.e., to generate a background area corresponding to the initial scene graph, and to obtain the scene graph of the product.

[0078] In an embodiment of the present application, a foreground image containing a target object and text information corresponding to the target object are received; description information and layout information of a target scene corresponding to the target object are determined based on the foreground image and text information by a target multimodal visual language model; a scene graph of the target object is generated based on the foreground image, description information and layout information by a target text graph generative model, wherein the target text graph generative model is obtained by training an initial text graph generative model based on a training sample set, and the target multimodal visual language model can generate a suitable scene graph description based on the object according to the knowledge learned by the target multimodal visual language model, thereby improving the fit between the scene and the object, improving the fit between the foreground and background scenes of the scene graph, and can also predict the object and the scene. The layout information of elements can effectively avoid problems such as unreasonable object size, inappropriate object placement, and unreasonable generation of scene elements; using the target text-generated image generation model to generate a scene graph according to the foreground image, description information and layout information, the object features can be effectively maintained, and the placement relationship between the target object and the surrounding objects is more in line with the actual objective laws, which improves the naturalness and availability of the image, and achieves the purpose of improving the quality of the raw image of the text-generated image generation model, thereby achieving the technical effect of improving the accuracy of the text description information and improving the visual effect of the generated scene graph, and further solves the technical problem that the scene graph of the object is generated based on the manually input text description information in the related technology, and the accuracy of the text description information is low, resulting in poor visual effect of the generated scene graph.

[0079] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0080] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0081] Example 2

[0082] According to an embodiment of the present application, a method for generating a target text graph generative model is also provided, such as Figure 5 As shown, the method includes:

[0083] Step S501, obtaining a training sample set, wherein the training sample set includes: a plurality of training sample pairs, the training sample pairs include a sample input and a sample output, the sample input is composed of a foreground image containing a sample object, a sample scene description and sample layout information, and the sample output is a sample scene graph corresponding to the sample input;

[0084] Step S502, adding a position information control module after the self-attention mechanism of the image generation network of the initial text image generation model to obtain a text image generation model to be trained, wherein the position information control module is used to control the image generation process of the image generation network according to the sample layout information;

[0085] Step S503, training the Vincent graph generative model to be trained according to the training sample set to obtain a trained Vincent graph generative model;

[0086] Step S504 , training the trained Vincent graph generative model according to the target control network to obtain a target Vincent graph generative model, wherein the target control network is used to control the object shape of the sample object.

[0087] Optionally, the initial image generation model uses the image generation model Stable Diffusion (SD for short). In order to keep the size and position of the object consistent with the prompt and ensure that the foreground features of the object are maintained, the SD model is modified, that is, after the self-attention mechanism of each layer of the U-net network (i.e., the image generation network) of the SD model, a position information control module is added to control the image generation process according to the layout information and guide the image generation model to generate the specified element at a given position. Fine-tune the modified model according to the training sample set, and add the target control network (inpaintingcontrolNet) for joint training in the later stage of training (i.e., after the trained image generation model is obtained), the image generation model (i.e., the target image generation model) that accepts the joint control of the foreground, prompt, and layout can be obtained. Among them, the target control network is used to keep the foreground shape of the product unchanged when the background is generated, that is, to control the foreground shape of the product from being affected by the background generation.

[0088] Example 3

[0089] According to an embodiment of the present application, a method for generating an image is also provided. Figure 6 As shown, the method includes:

[0090] Step S601, obtaining a foreground image containing a target object and text information corresponding to the target object uploaded by a client;

[0091] Step S602, in the cloud server, the description information and layout information of the target scene corresponding to the target object are determined by the target multimodal visual language model according to the foreground image and text information; the scene graph of the target object is generated by the target text graph generative model according to the foreground image, the description information and the layout information, wherein the target text graph generative model is obtained by training the initial text graph generative model according to the training sample set;

[0092] Step S603: Feedback the scene graph of the target object to the client.

[0093] Through the above scheme, the target multimodal visual language model can generate a suitable scene graph description according to the object based on the knowledge it has learned, thereby improving the fit between the scene and the object, and improving the fit between the foreground and background scenes of the object in the scene graph. It can also predict the layout information of the object and the scene elements, and effectively avoid problems such as unreasonable object size, inappropriate object placement, and unreasonable generation of scene elements; using the target text-generated image generation model to generate a scene graph based on the foreground image, description information, and layout information, the object features can be effectively maintained, and the placement relationship between the target object and the surrounding objects is more in line with the actual objective laws, which improves the naturalness and availability of the image, and achieves the purpose of improving the quality of the raw image of the text-generated image generation model, thereby achieving the technical effect of improving the accuracy of the text description information and improving the visual effect of the generated scene graph, and then solving the technical problem that the scene graph of the object is generated based on the manually input text description information in the related technology, and the accuracy of the text description information is low, resulting in poor visual effect of the generated scene graph.

[0094] In the cloud server, the specific method for generating the image is the same as that in the first embodiment, and will not be described again here.

[0095] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0097] Example 4

[0098] According to an embodiment of the present application, an image generating device for implementing the above-mentioned image generating method is also provided. Figure 7 As shown, the device includes: a receiving unit 701, a determining unit 702, and a processing unit 703.

[0099] The receiving unit 701 is used to receive a foreground image containing a target object and text information corresponding to the target object;

[0100] A determination unit 702 is used to determine description information and layout information of a target scene corresponding to a target object according to the foreground image and text information through a target multimodal visual language model;

[0101] The processing unit 703 is used to generate a scene graph of the target object according to the foreground image, description information and layout information through a target text graph generative model, wherein the target text graph generative model is obtained by training the initial text graph generative model according to the training sample set.

[0102] In the image generation device provided in the fourth embodiment of the present application, a foreground image containing a target object and text information corresponding to the target object are received by a receiving unit 701; a determination unit 702 determines description information and layout information of a target scene corresponding to the target object based on the foreground image and the text information through a target multimodal visual language model; a processing unit 703 generates a scene graph of the target object based on the foreground image, the description information and the layout information through a target text graph generative model, wherein the target text graph generative model is obtained by training an initial text graph generative model based on a training sample set. In this scheme, the target multimodal visual language model can generate a suitable scene graph description according to the object based on the knowledge it has learned, thereby improving the fit between the scene and the object, and improving the fit between the foreground and background scenes of the object in the scene graph. It can also predict the layout information of the object and the scene elements, and effectively avoid problems such as unreasonable object size, inappropriate object placement, and unreasonable generation of scene elements; the target text-generated graph generation model is used to generate a scene graph based on the foreground image, description information, and layout information, so that the object features can be effectively maintained, and the placement relationship between the target object and the surrounding objects is more in line with the actual objective laws, which improves the naturalness and availability of the image, and achieves the purpose of improving the quality of the raw image of the text-generated graph generation model, thereby achieving the technical effect of improving the accuracy of the text description information and improving the visual effect of the generated scene graph, and then solving the technical problem that the scene graph of the object is generated based on the manually input text description information in the related technology, and the accuracy of the text description information is low, resulting in poor visual effect of the generated scene graph.

[0103] Optionally, in the image generation device provided in Example 4 of the present application, the determination unit includes: a first determination subunit, used to determine the description information based on the foreground image and text information through a first model, wherein the first model is obtained by training the scene description output capability of the initial multimodal visual language model based on the first training sample set; a second determination subunit, used to determine the layout information based on the foreground image and the description information through a second model, wherein the second model is obtained by training the layout information generation capability of the initial multimodal visual language model based on the second training sample set.

[0104] Optionally, in the image generation device provided in Example 4 of the present application, the first determination subunit includes: a first processing module, used to process the foreground image and text information respectively through a first model to obtain an image feature representation corresponding to the foreground image and a text feature representation corresponding to the text information; a second processing module, used to process the image feature representation and the text feature representation based on a self-attention mechanism to obtain a fused feature representation, and predict the description information based on the fused feature representation.

[0105] Optionally, in the image generating device provided in Example 4 of the present application, the second determining subunit includes: a third processing module, used to extract the target object from the foreground image through the second model, and extract multiple background elements from the description information; a fourth processing module, used to perform size and position prediction based on the target object and multiple background elements, and obtain the size and position information of the target object, as well as the size and position information of the background elements among the multiple background elements; a fifth processing module, used to compose layout information based on the size and position information of the target object, as well as the size and position information of the background elements among the multiple background elements.

[0106] Optionally, in the image generation device provided in Example 4 of the present application, the processing unit includes: a first processing sub-unit, used to extract the target object from the foreground image through a target text image generation model, and adjust the size and position of the target object according to the size and position information of the target object in the layout information to obtain an initial scene graph; a second processing sub-unit, used to control the foreground pixels of the target object in the initial scene graph to be fixed according to the description information and layout information, generate a background area corresponding to the initial scene graph, and obtain a scene graph of the target object.

[0107] Optionally, in the image generation device provided in Example 4 of the present application, the second processing sub-unit includes: a sixth processing module, which is used to process the background elements in the layout information through the target text image generation model to obtain a mark embedding vector, and process the position information of the background elements in the multiple background elements in the layout information to obtain a position embedding vector; a seventh processing module, which is used to splice the mark embedding vector and the position embedding vector to obtain a target embedding vector; and an eighth processing module, which is used to input the target embedding vector into the position information control module of the target text image generation model, so that the position information control module generates a background area corresponding to the initial scene graph according to the target embedding vector to obtain a scene graph of the target object.

[0108] It should be noted that the above-mentioned receiving unit 701, determining unit 702, and processing unit 703 correspond to steps S201 to S203 in Example 1, and the above-mentioned units and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.

[0109] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0110] Example 5

[0111] According to an embodiment of the present application, a device for generating a target text graph generative model for implementing the above-mentioned method for generating a target text graph generative model is also provided. Figure 8 As shown, the device includes: an acquisition unit 801, a construction unit 802, a first training unit 803, and a second training unit 804.

[0112] The acquisition unit 801 is used to acquire a training sample set, wherein the training sample set includes: a plurality of training sample pairs, the training sample pairs include a sample input and a sample output, the sample input includes a foreground image including a sample object, a sample scene description and sample layout information, and the sample output is a sample scene graph corresponding to the sample input;

[0113] A construction unit 802 is used to add a position information control module after the self-attention mechanism of the image generation network of the initial text image generation model to obtain a text image generation model to be trained, wherein the position information control module is used to control the image generation process of the image generation network according to the sample layout information;

[0114] A first training unit 803 is used to train the Vincent graph generative model to be trained according to the training sample set to obtain a trained Vincent graph generative model;

[0115] The second training unit 804 is used to train the trained Vincent graph generative model according to the target control network to obtain a target Vincent graph generative model, wherein the target control network is used to control the object shape of the sample object.

[0116] It should be noted here that the above-mentioned acquisition unit 801, construction unit 802, first training unit 803, and second training unit 804 correspond to steps S501 to S504 in Example 2, and the instances and application scenarios implemented by the above-mentioned units and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 2.

[0117] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 2 as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 2.

[0118] Example 6

[0119] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0120] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0121] In this embodiment, the electronic device can execute the program code of the following steps in the image generation method: receiving a foreground image containing a target object and text information corresponding to the target object; determining the description information and layout information of the target scene corresponding to the target object based on the foreground image and the text information through a target multimodal visual language model; generating a scene graph of the target object based on the foreground image, the description information and the layout information through a target text graph generative model, wherein the target text graph generative model is obtained by training an initial text graph generative model based on a training sample set.

[0122] The electronic device can also execute the program code of the following steps in the image generation method: determining description information based on the foreground image and text information through a first model, wherein the first model is obtained by training the scene description output capability of the initial multimodal visual language model based on a first training sample set; determining layout information based on the foreground image and description information through a second model, wherein the second model is obtained by training the layout information generation capability of the initial multimodal visual language model based on a second training sample set.

[0123] The above-mentioned electronic device can also execute the program code of the following steps in the image generation method: respectively processing the foreground image and text information through the first model to obtain the image feature representation corresponding to the foreground image and the text feature representation corresponding to the text information; processing the image feature representation and the text feature representation based on the self-attention mechanism to obtain a fused feature representation, and predicting the description information based on the fused feature representation.

[0124] The above-mentioned electronic device can also execute the program code of the following steps in the image generation method: extracting the target object from the foreground image through the second model, and extracting multiple background elements from the description information; predicting the size and position based on the target object and the multiple background elements to obtain the size and position information of the target object, as well as the size and position information of the background elements among the multiple background elements; and forming layout information based on the size and position information of the target object, as well as the size and position information of the background elements among the multiple background elements.

[0125] The above-mentioned electronic device can also execute the program code of the following steps in the image generation method: extracting the target object from the foreground image through the target text image generation model, and adjusting the size and position of the target object according to the size and position information of the target object in the layout information to obtain the initial scene graph; controlling the foreground pixels of the target object in the initial scene graph to be fixed according to the description information and layout information, generating the background area corresponding to the initial scene graph, and obtaining the scene graph of the target object.

[0126] The electronic device can also execute the program code of the following steps in the image generation method: processing the background elements in the layout information through the target text image generation model to obtain the mark embedding vector, and processing the position information of the background elements in the multiple background elements in the layout information to obtain the position embedding vector; splicing the mark embedding vector and the position embedding vector to obtain the target embedding vector; inputting the target embedding vector into the position information control module of the target text image generation model, so that the position information control module generates the background area corresponding to the initial scene graph according to the target embedding vector, and obtains the scene graph of the target object.

[0127] Optionally, Fig. 9 is a structural block diagram of an electronic device according to an embodiment of the present application. Fig. 9 As shown, the electronic device 90 may include: one or more ( Fig. 9 Only one is shown in the figure) processor 902, memory 904. The electronic device 90 may also include a storage controller, through which the memory 904 is controlled and managed; the electronic device 90 may also include a peripheral interface, through which the radio frequency module, audio module and display screen are connected.

[0128] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the image generation method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned image generation method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0129] The processor can call the information and application stored in the memory through the transmission device to execute the following steps: receiving a foreground image containing a target object and text information corresponding to the target object; determining the description information and layout information of the target scene corresponding to the target object based on the foreground image and the text information through a target multimodal visual language model; generating a scene graph of the target object based on the foreground image, the description information and the layout information through a target text graph generative model, wherein the target text graph generative model is obtained by training an initial text graph generative model based on a training sample set.

[0130] Optionally, the processor may also execute program code of the following steps: determining description information based on the foreground image and text information through a first model, wherein the first model is obtained by training the scene description output capability of the initial multimodal visual language model based on a first training sample set; determining layout information based on the foreground image and description information through a second model, wherein the second model is obtained by training the layout information generation capability of the initial multimodal visual language model based on a second training sample set.

[0131] Optionally, the processor may also execute the program code of the following steps: processing the foreground image and text information respectively through the first model to obtain the image feature representation corresponding to the foreground image and the text feature representation corresponding to the text information; processing the image feature representation and the text feature representation based on the self-attention mechanism to obtain a fused feature representation, and predicting the description information based on the fused feature representation.

[0132] Optionally, the processor may also execute the program code of the following steps: extracting a target object from the foreground image through a second model, and extracting multiple background elements from the description information; predicting the size and position of the target object and the multiple background elements to obtain the size and position information of the target object, as well as the size and position information of the background elements among the multiple background elements; and composing layout information based on the size and position information of the target object, as well as the size and position information of the background elements among the multiple background elements.

[0133] Optionally, the processor may also execute the program code of the following steps: extracting a target object from a foreground image through a target scene graph generation model, and adjusting the size and position of the target object according to the size and position information of the target object in the layout information to obtain an initial scene graph; controlling the foreground pixels of the target object in the initial scene graph to be fixed according to the description information and the layout information, generating a background area corresponding to the initial scene graph, and obtaining a scene graph of the target object.

[0134] Optionally, the processor may also execute the program code of the following steps: processing the background elements in the layout information through the target text image generation model to obtain a mark embedding vector, and processing the position information of the background elements in the multiple background elements in the layout information to obtain a position embedding vector; concatenating the mark embedding vector and the position embedding vector to obtain a target embedding vector; inputting the target embedding vector into the position information control module of the target text image generation model, so that the position information control module generates a background area corresponding to the initial scene graph according to the target embedding vector, and obtains the scene graph of the target object.

[0135] It can be understood by those skilled in the art that Fig. 9 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Fig. 9 The structure of the electronic device is not limited. Fig. 9 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Fig. 9 Different configurations shown.

[0136] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0137] Example 7

[0138] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image generation method provided in the first embodiment.

[0139] Optionally, in this embodiment, the storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.

[0140] Example 8

[0141] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and when the computer program is executed by a processor, the image generation method provided in the first embodiment is implemented.

[0142] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0143] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0148] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image generation method, characterized in that: include: Receiving a foreground image containing a target object and text information corresponding to the target object; Determining description information and layout information of a target scene corresponding to the target object according to the foreground image and the text information through a target multimodal visual language model; Generate a scene graph of the target object according to the foreground image, the description information and the layout information by using a target text graph generative model, wherein the target text graph generative model is obtained by training an initial text graph generative model according to a training sample set; The target multimodal visual language model includes a first model and a second model, the input of the first model is the foreground image and the text information, the output of the first model is the description information, the input of the second model is the foreground image and the description information, and the output of the second model is the layout information; the input of the target text image generation model is the foreground image, the description information and the layout information, and the output of the target text image generation model is the scene graph of the target object; the training sample set includes: a plurality of training sample pairs, the training sample pairs include a sample input and a sample output, the sample input consists of a foreground image containing a sample object, a sample scene description and a sample layout information, and the sample output is a sample scene graph corresponding to the sample input; The target VGG generative model is obtained by training the initial VGG generative model based on the training sample set, including: adding a position information control module after the self-attention mechanism of the image generation network of the initial VGG generative model to obtain a modified model, fine-tuning the modified model based on the training sample set, and adding a control network for joint training in the later stage of training to obtain the target VGG generative model, wherein the position information control module is used to control the image generation process based on layout information.

2. The method according to claim 1, characterized in that Determining the description information and layout information of the target scene corresponding to the target object according to the foreground image and the text information through the target multimodal visual language model includes: Determining the description information according to the foreground image and the text information through the first model, wherein the first model is obtained by training the scene description output capability of the initial multimodal visual language model according to the first training sample set; The layout information is determined by the second model according to the foreground image and the description information, wherein the second model is obtained by training the layout information generation capability of the initial multimodal visual language model according to a second training sample set.

3. The method according to claim 2, characterized in that Determining the description information by the first model according to the foreground image and the text information includes: Processing the foreground image and the text information respectively by using the first model to obtain an image feature representation corresponding to the foreground image and a text feature representation corresponding to the text information; The image feature representation and the text feature representation are processed based on a self-attention mechanism to obtain a fused feature representation, and the description information is predicted based on the fused feature representation.

4. The method according to claim 2, characterized in that: Determining the layout information according to the foreground image and the description information through the second model includes: extracting the target object from the foreground image by using the second model, and extracting a plurality of background elements from the description information; Predicting the size and position of the target object and the multiple background elements to obtain the size and position information of the target object and the size and position information of the background elements in the multiple background elements; The layout information is composed according to the size and position information of the target object and the size and position information of the background elements among the multiple background elements.

5. The method according to claim 4, characterized in that Generating the scene graph of the target object according to the foreground image, the description information and the layout information by using the target text graph generative model includes: Extracting the target object from the foreground image through the target scene graph generation model, and adjusting the size and position of the target object according to the size and position information of the target object in the layout information to obtain an initial scene graph; The foreground pixels of the target object in the initial scene graph are controlled to be fixed according to the description information and the layout information, a background area corresponding to the initial scene graph is generated, and a scene graph of the target object is obtained.

6. The method according to claim 5, characterized in that Generating a background area corresponding to the initial scene graph to obtain the scene graph of the target object includes: Processing the background elements in the layout information by the target text graph generative model to obtain a label embedding vector, and processing the position information of the background elements in the multiple background elements in the layout information to obtain a position embedding vector; Concatenate the tag embedding vector and the position embedding vector to obtain a target embedding vector; The target embedding vector is input into the position information control module of the target text image generation model, so that the position information control module generates a background area corresponding to the initial scene image according to the target embedding vector to obtain the scene image of the target object.

7. A method for generating a target text graph generative model, characterized in that: include: Acquire a training sample set, wherein the training sample set includes: a plurality of training sample pairs, the training sample pairs include a sample input and a sample output, the sample input is composed of a foreground image containing a sample object, a sample scene description, and sample layout information, and the sample output is a sample scene image corresponding to the sample input; After the self-attention mechanism of the image generation network of the initial text image generation model, a position information control module is added to obtain the text image generation model to be trained, wherein the position information control module is used to control the image generation process of the image generation network according to the sample layout information; Training the to-be-trained Vincent graph generative model according to the training sample set to obtain a trained Vincent graph generative model; The trained Vincent graph generative model is trained according to a target control network to obtain the target Vincent graph generative model according to any one of claims 1 to 6, wherein the target control network is used to control the object shape of the sample object.

8. An image generation method, characterized in that: include: Obtaining a foreground image containing a target object and text information corresponding to the target object uploaded by a client; Determining, in the cloud server, description information and layout information of a target scene corresponding to the target object based on the foreground image and the text information using a target multimodal visual language model; Generate a scene graph of the target object according to the foreground image, the description information and the layout information by using a target text graph generative model, wherein the target text graph generative model is obtained by training an initial text graph generative model according to a training sample set; Feeding back the scene graph of the target object to the client; The target multimodal visual language model includes a first model and a second model, the input of the first model is the foreground image and the text information, the output of the first model is the description information, the input of the second model is the foreground image and the description information, and the output of the second model is the layout information; the input of the target text image generation model is the foreground image, the description information and the layout information, and the output of the target text image generation model is the scene graph of the target object; the training sample set includes: a plurality of training sample pairs, the training sample pairs include a sample input and a sample output, the sample input consists of a foreground image containing a sample object, a sample scene description and a sample layout information, and the sample output is a sample scene graph corresponding to the sample input; The target VGG generative model is obtained by training the initial VGG generative model based on the training sample set, including: adding a position information control module after the self-attention mechanism of the image generation network of the initial VGG generative model to obtain a modified model, fine-tuning the modified model based on the training sample set, and adding a control network for joint training in the later stage of training to obtain the target VGG generative model, wherein the position information control module is used to control the image generation process based on layout information.

9. An image generating device, characterized in that: include: A receiving unit, configured to receive a foreground image containing a target object and text information corresponding to the target object; A determination unit, configured to determine description information and layout information of a target scene corresponding to the target object according to the foreground image and the text information through a target multimodal visual language model; A processing unit, configured to generate a scene graph of the target object according to the foreground image, the description information and the layout information by using a target text graph generative model, wherein the target text graph generative model is obtained by training an initial text graph generative model according to a training sample set; The target multimodal visual language model includes a first model and a second model, the input of the first model is the foreground image and the text information, the output of the first model is the description information, the input of the second model is the foreground image and the description information, and the output of the second model is the layout information; the input of the target text image generation model is the foreground image, the description information and the layout information, and the output of the target text image generation model is the scene graph of the target object; the training sample set includes: a plurality of training sample pairs, the training sample pairs include a sample input and a sample output, the sample input consists of a foreground image containing a sample object, a sample scene description and a sample layout information, and the sample output is a sample scene graph corresponding to the sample input; The target VGG generative model is obtained by training the initial VGG generative model based on the training sample set, including: adding a position information control module after the self-attention mechanism of the image generation network of the initial VGG generative model to obtain a modified model, fine-tuning the modified model based on the training sample set, and adding a control network for joint training in the later stage of training to obtain the target VGG generative model, wherein the position information control module is used to control the image generation process based on layout information.

10. A device for generating a target text graph generative model, characterized in that: include: An acquisition unit is used to acquire a training sample set, wherein the training sample set includes: a plurality of training sample pairs, the training sample pairs include a sample input and a sample output, the sample input is composed of a foreground image containing a sample object, a sample scene description and sample layout information, and the sample output is a sample scene graph corresponding to the sample input; A construction unit is used to add a position information control module after the self-attention mechanism of the image generation network of the initial text image generation model to obtain a text image generation model to be trained, wherein the position information control module is used to control the image generation process of the image generation network according to the sample layout information; A first training unit is used to train the to-be-trained Vincent graph generative model according to the training sample set to obtain a trained Vincent graph generative model; A second training unit is used to train the trained Vincent graph generative model according to a target control network to obtain the target Vincent graph generative model according to any one of claims 1 to 6, wherein the target control network is used to control the object shape of the sample object.

11. An electronic device, characterized in that: include: A memory storing an executable program; A processor, used to run the program, wherein the program executes the image generation method of any one of claims 1 to 6 and 8 when running.

12. A computer program product, characterized in that The method comprises a computer program or instructions, which implement the image generation method according to any one of claims 1 to 6 and 8 when executed by a processor.

Images