Image generation method, target cultural image large model generation method and device

Abstract

The present application discloses an image generation method, a method and device for generating a target Wensheng graph large model. The method relates to the field of artificial intelligence, and includes: obtaining target object information and attribute information corresponding to the target object information in text information through a target Wensheng graph large model; determining a mask image corresponding to the target object information, and processing the output information of a target visual control network based on the mask image to obtain first image information; generating a target image corresponding to the text information based on the first image information, the target object information, the attribute information corresponding to the target object information, and a target loss function corresponding to the target Wensheng graph large model. The present application solves the technical problem that a Wensheng graph large model in the related art performs attribute binding on an image to be generated under local visual control conditions, resulting in blurred image background and low attribute binding accuracy, resulting in poor generated image effects.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to an image generation method, a method and a device for generating a large model of a target cultural image. Background Art

[0002] In the context of text-generated images, in order to control the visual effect, users usually add attribute constraints to the image they want to generate in the text prompt, such as a yellow sofa. However, due to the low overlap between the attributes and the attention map of the object, the current text-generated image model cannot effectively generate the specified attributes.

[0003] The existing training-free methods for solving the problem of low overlap between the attention maps of attributes and objects have some problems when there is local visual control: the background will be blurred due to the influence of local control and often cannot comply with the prompt control; the attribute binding accuracy is low, the failure rate is high, and it is easy to violate the control of visual conditions, resulting in unwanted results for users and poor generated image effects.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide an image generation method, a generation method and device of a target Vincent graph large model, so as to at least solve the technical problem that the Vincent graph large model in the related art performs attribute binding on the image to be generated under local visual control conditions, resulting in blurred image background and low attribute binding accuracy, resulting in poor generated image effect.

[0006] According to one aspect of an embodiment of the present application, an image generation method is provided, including: obtaining target object information and attribute information corresponding to the target object information in text information through a target Wensheng graph large model, wherein the target Wensheng graph large model is obtained by training an initial Wensheng graph large model based on the attribute information and position information of the object in a training set image; determining a mask image corresponding to the target object information, and processing output information of a target visual control network based on the mask image to obtain first image information, wherein the target visual control network is used to control the object shape corresponding to the target object information, and the first image information is used to indicate a visual control area corresponding to the target object information; generating a target image corresponding to the text information based on the first image information, the target object information, the attribute information corresponding to the target object information, and a target loss function corresponding to the target Wensheng graph large model, wherein the target loss function is used to limit the attribute binding between the target object information and the attribute information corresponding to the target object information within the visual control area.

[0007] Furthermore, the output information of the target visual control network is processed based on the mask image to obtain the first image information, including: obtaining a line drawing image corresponding to the target object information; calling the target visual control network to process the line drawing image to obtain the output information of the target visual control network, wherein the output information is used to indicate contour information corresponding to the target object information; and performing information filtering processing on the output information based on the mask image to obtain the first image information.

[0008] Furthermore, generating a target image corresponding to the text information based on the first image information, the target object information, the attribute information corresponding to the target object information, and the target loss function corresponding to the target text image model includes: inputting the first image information, the target object information, and the attribute information corresponding to the target object information into a target network module; using the mask image as a constraint condition of the target loss function, iteratively updating the target network module according to the target loss function until a preset number of iterations is reached to obtain the target image corresponding to the text information.

[0009] Furthermore, the target loss function includes a first loss function and a second loss function, the first loss function and the second loss function correspond to different targets, and iteratively updating the target network module according to the target loss function includes: obtaining an attention map of the target object information and an attention map of the attribute information corresponding to the target object information; calculating a divergence distance between the attention map of the target object information and the attention map of the attribute information corresponding to the target object information according to the first loss function; determining a focus area of ​​the attention map of the target object information and a focus area of ​​the attention map of the attribute information corresponding to the target object information; calculating the similarity between the focus area of ​​the attention map of the target object information and the focus area of ​​the attention map of the attribute information corresponding to the target object information according to the second loss function; and iteratively updating the target network module according to the divergence distance and the similarity.

[0010] Furthermore, iteratively updating the target network module according to the divergence distance and the similarity includes: iteratively updating the target network module with the divergence distance reaching a preset distance as a goal, and with the similarity reaching a preset similarity as a goal.

[0011] Furthermore, obtaining target object information and attribute information corresponding to the target object information in text information through the target text-graph large model includes: receiving text information input by the target object through the target text-graph large model; calling a natural language processing tool to perform syntactic analysis on the text information to obtain target object information and attribute information corresponding to the target object information.

[0012] According to another aspect of an embodiment of the present application, a method for generating a target large model of a text graph is provided, comprising: obtaining a training set, wherein the training set includes a plurality of image-text pair samples; adding a position network module before the self-attention mechanism of the middle layer of the initial large model of the text graph to obtain a large model of the text graph to be trained, wherein the position network module is used to inject attribute information and position information of objects in the image in the image-text pair samples; and training the large model of the text graph to be trained according to the training set to obtain a target large model of the text graph.

[0013] According to another aspect of an embodiment of the present application, an image generation method is also provided, including: obtaining text information uploaded by a client; obtaining target object information and attribute information corresponding to the target object information in the text information through a target Wensheng graph large model in a cloud server, wherein the target Wensheng graph large model is obtained by training an initial Wensheng graph large model based on the attribute information and position information of the object in the training set image; determining a mask image corresponding to the target object information, and processing output information of a target visual control network based on the mask image to obtain first image information, wherein the target visual control network is used to control the object shape corresponding to the target object information, and the first image information is used to indicate a visual control area corresponding to the target object information; generating a target image corresponding to the text information based on the first image information, the target object information, the attribute information corresponding to the target object information, and the target loss function corresponding to the target Wensheng graph large model, wherein the target loss function is used to limit the attribute binding between the target object information and the attribute information corresponding to the target object information within the visual control area; and feeding back the target image to the client.

[0014] According to another aspect of an embodiment of the present application, an image generating device is also provided, including: a first acquiring unit, used to acquire target object information and attribute information corresponding to the target object information in text information through a target Wensheng graph large model, wherein the target Wensheng graph large model is obtained by training an initial Wensheng graph large model based on the attribute information and position information of the object in the training set image; a first processing unit, used to determine a mask image corresponding to the target object information, and to process output information of a target visual control network based on the mask image to obtain first image information, wherein the target visual control network is used to control the object shape corresponding to the target object information, and the first image information is used to indicate a visual control area corresponding to the target object information; a second processing unit, used to generate a target image corresponding to the text information based on the first image information, the target object information, the attribute information corresponding to the target object information, and the target loss function corresponding to the target Wensheng graph large model, wherein the target loss function is used to limit the attribute binding between the target object information and the attribute information corresponding to the target object information within the visual control area.

[0015] Furthermore, the first processing unit includes: an acquisition subunit, used to acquire a line draft image corresponding to the target object information; a first processing subunit, used to call the target visual control network to process the line draft image to obtain output information of the target visual control network, wherein the output information is used to indicate contour information corresponding to the target object information; a second processing subunit, used to perform information filtering processing on the output information based on the mask image to obtain first image information.

[0016] Furthermore, the second processing unit includes: a third processing sub-unit, used to input the first image information, the target object information and the attribute information corresponding to the target object information into the target network module; a fourth processing sub-unit, used to use the mask image as the constraint condition of the target loss function, and iteratively update the target network module according to the target loss function until a preset number of iterations is reached to obtain the target image corresponding to the text information.

[0017] Furthermore, the target loss function includes a first loss function and a second loss function, the first loss function and the second loss function correspond to different targets, and the fourth processing subunit includes: an acquisition module, used to acquire the attention map of the target object information and the attention map of the attribute information corresponding to the target object information; a first calculation module, used to calculate the divergence distance between the attention map of the target object information and the attention map of the attribute information corresponding to the target object information based on the first loss function; a determination module, used to determine the focus area of ​​the attention map of the target object information and the focus area of ​​the attention map of the attribute information corresponding to the target object information; a second calculation module, used to calculate the similarity between the focus area of ​​the attention map of the target object information and the focus area of ​​the attention map of the attribute information corresponding to the target object information based on the second loss function; a processing module, used to iteratively update the target network module based on the divergence distance and the similarity.

[0018] Furthermore, the processing module includes: a processing submodule, which is used to iteratively update the target network module with the divergence distance reaching a preset distance as a goal and the similarity reaching a preset similarity as a goal.

[0019] Furthermore, the first acquisition unit includes: a receiving subunit, which is used to receive text information input by the target object through the target text image model; an analyzing subunit, which is used to call a natural language processing tool to perform syntactic analysis on the text information to obtain target object information and attribute information corresponding to the target object information.

[0020] According to another aspect of an embodiment of the present application, a device for generating a target Wensheng graph large model is also provided, including: a second acquisition unit, used to acquire a training set, wherein the training set includes multiple image-text pair samples; a third processing unit, used to add a position network module before the self-attention mechanism of the middle layer of the initial Wensheng graph large model to obtain a Wensheng graph large model to be trained, wherein the position network module is used to inject attribute information and position information of objects in the image in the image-text pair samples; a fourth processing unit, used to train the Wensheng graph large model to be trained based on the training set to obtain a target Wensheng graph large model.

[0021] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein any one of the above-mentioned image generation methods is executed when the program is run.

[0022] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a program, wherein when the program is running, the device where the storage medium is located is controlled to execute any one of the above-mentioned image generation methods.

[0023] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program or instructions, and when the computer program or instructions are executed by a processor, any one of the above-mentioned image generation methods is implemented.

[0024] In an embodiment of the present application, target object information in text information and attribute information corresponding to the target object information are obtained through a target text image large model, wherein the target text image large model is obtained by training an initial text image large model based on the attribute information and position information of the object in the training set image; a mask image corresponding to the target object information is determined, and the output information of the target visual control network is processed based on the mask image to obtain first image information, wherein the target visual control network is used to control the object shape corresponding to the target object information, and the first image information is used to indicate the visual control area corresponding to the target object information; a target image corresponding to the text information is generated based on the first image information, the target object information, the attribute information corresponding to the target object information, and the target loss function corresponding to the target text image large model, wherein the target loss function is used to limit the attribute binding between the target object information and the attribute information corresponding to the target object information to the visual control area In this way, the output information of the target visual control network is processed based on the mask image, so as to realize the shape control of the object within the mask and avoid the interference of the visual control on the image generation outside the mask. Through the target loss function corresponding to the target Vincent graph large model, it is realized under the dual constraints of visual control and object mask, the distance between the attention map of the object and its corresponding attribute is shortened, the alignment of the attention map and the control of local visual conditions are taken into account, the instruction following ability of the Vincent graph large model is improved, and accurate background generation and attribute binding are realized. The purpose of realizing the attribute binding between the object and its corresponding attribute while maintaining the accurate control of the visual conditions is achieved, thereby realizing the technical effect of improving the effect of the generated image, and further solving the technical problem that the Vincent graph large model in the related art performs attribute binding on the generated image under the local visual control condition, and the image background is blurred and the attribute binding accuracy is low, resulting in poor generated image effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0026] Figure 1 is a schematic diagram of a computer terminal provided according to Embodiment 1 of the present application;

[0027] Figure 2 is a flowchart of an image generation method provided according to Embodiment 1 of the present application;

[0028] Figure 3 It is a schematic diagram of an optional text-to-image generation process provided according to the first embodiment of the present application;

[0029] Figure 4It is a flowchart of a method for generating a target cultural graph model according to the second embodiment of the present application;

[0030] Figure 5 is a flowchart of an image generation method provided according to Embodiment 3 of the present application;

[0031] Figure 6 is a schematic diagram of an image generating device provided according to Embodiment 4 of the present application;

[0032] Figure 7 It is a schematic diagram of a device for generating a target cultural graph large model according to the fifth embodiment of the present application;

[0033] Figure 8 This is a structural block diagram of an electronic device provided according to Embodiment 6 of the present application. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0037] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0038] Instruction following: According to the text instructions, an image that meets the description is generated based on the Wenshengtu model.

[0039] Attribute binding: When generating an image based on text, the attributes of the objects in the image (such as color, shape) match the corresponding description in the text.

[0040] Example 1

[0041] According to an embodiment of the present application, an image generating method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image generation method. Figure 1 As shown, the computer terminal (or mobile device) 10 may include a processor set 102 (the processor set 102 may include but is not limited to a processing device such as a microprocessor (Microcontroller Unit, referred to as MCU) or a programmable logic device (Field Programmable Gate Array, referred to as FPGA), and the processor set 102 may include a processor set, Figure 1 102a, 102b, ..., 102n are used to illustrate), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0043] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuits may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device).

[0044] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the image generation method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned image generation method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0045] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0046] The display may be a touch screen type liquid crystal display, which may enable a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0047] In the context of text-generated images, in order to control the visual effect, users usually add attribute constraints to the image they want to generate in the prompt, such as a yellow sofa. However, due to the low overlap between the attributes and the attention map of the object, the current text-generated image model cannot effectively generate the specified attributes.

[0048] The existing training-free methods for solving the problem of low overlap between the attention maps of attributes and objects have some problems when there is local visual control: the background will be blurred due to the influence of local control and often cannot comply with the prompt control; the attribute binding accuracy is low, the failure rate is high, and it is easy to violate the control of visual conditions, resulting in unwanted results for users and poor generated image effects.

[0049] In the above technical background, the present application provides Figure 2 The image generation method shown. Figure 2 : is a flow chart of an image generation method provided according to Embodiment 1 of the present application. The method comprises:

[0050] Step S201, obtaining target object information in text information and attribute information corresponding to the target object information through a target text image large model, wherein the target text image large model is obtained by training an initial text image large model based on the attribute information and position information of the object in the training set image.

[0051] Optionally, the initial text-based image model uses the text-based image model Stable Diffusion (SD for short). Based on the training-free solution, although low-cost object position control and attribute binding can be achieved, since the attention map needs to be modified during the forward denoising process, this behavior is not aligned with the SD model training process, which also causes problems such as unnatural images and insufficient aesthetics to a certain extent, and is slightly insufficient for application in real scenes. Therefore, the image-text pair samples and the position information of significant objects in the image can be collected to construct a training set for training object generation under layout control.

[0052] For example, a position network module (position net) is added before the self-attention mechanism (self-attention) of the middle block layer of the U-net network of the SD model to inject the attributes and position information of the objects in the image. During training, other network parameters are frozen, and only the newly added position net module is fine-tuned to train the initial text image model according to the attribute information and position information of the objects in the training set images to obtain the target text image model, so that the trained text image model (i.e., the target text image model) has better control over the position and attribute binding of objects, and the resulting image is more natural and beautiful, and can be more adaptable to complex scenes.

[0053] Therefore, the trained text graph model is used to obtain the target object information in the text information (i.e., prompt) and the attribute information corresponding to the target object information. For example, the prompt input by the user is: a yellow dog, a green skateboard, and huge rocks on the mountaintop. The target object information can be the object words in the prompt, such as dog and skateboard, and the corresponding attribute information is the attribute words in the prompt, such as yellow and green.

[0054] Step S202, determine the mask image corresponding to the target object information, and process the output information of the target visual control network based on the mask image to obtain first image information, wherein the target visual control network is used to control the object shape corresponding to the target object information, and the first image information is used to indicate the visual control area corresponding to the target object information.

[0055] Optionally, after acquiring the target object information, a mask image corresponding to the target object information can be determined, for example, a mask image corresponding to dog and a mask image corresponding to skateboard can be determined, and then the output information of the target visual control network is processed based on the mask image to obtain the first image information, wherein the target visual control network can be a visual control network (ControlNet) combined with an edge detection model (Canny model), namely, Canny ControlNet, and the output information of Canny ControlNet can be processed by masking the part that does not need to be controlled at the output layer of Canny ControlNet, namely, filtering the output information of Canny ControlNet to obtain the area that needs visual control (namely, the first image information).

[0056] Step S203, generating a target image corresponding to the text information based on the first image information, the target object information, the attribute information corresponding to the target object information, and the target loss function corresponding to the target text image model, wherein the target loss function is used to limit the attribute binding between the target object information and the attribute information corresponding to the target object information within the visual control area.

[0057] Optionally, a target loss function is constructed based on the attribute matching loss, and the attention maps of the object words and the corresponding attribute words are gradually guided to focus on the correct area, so as to bind the attribute to the corresponding object without conflicting with the visual control condition. Based on the area that needs to be visually controlled, the object words, the attribute words corresponding to the object words, and the target loss function, an image corresponding to the prompt (i.e., the target image) can be generated.

[0058] In this scheme, the output information of the target visual control network is processed based on the mask image, so as to realize the shape control of the object within the mask and avoid the interference of visual control on the image generation outside the mask. Through the target loss function corresponding to the target Vincent graph large model, it is realized that under the dual constraints of visual control and object mask, the distance between the attention map of the object and its corresponding attribute is shortened, and the alignment of the attention map and the control of local visual conditions are taken into account, thereby improving the instruction following ability of the Vincent graph large model, realizing accurate background generation and attribute binding, and achieving the purpose of attribute binding between the object and its corresponding attribute while maintaining accurate control of visual conditions, thereby achieving the technical effect of improving the effect of the generated image, and further solving the technical problem that the Vincent graph large model in the related art performs attribute binding on the generated image under local visual control conditions, and the image background is blurred and the attribute binding accuracy is low, resulting in poor generated image effect.

[0059] In order to effectively control the edge contour of an object, in the image generation method provided in Example 1 of the present application, the output information of the target visual control network is processed based on the mask image to obtain the first image information, including: obtaining a line draft image corresponding to the target object information; calling the target visual control network to process the line draft image to obtain the output information of the target visual control network, wherein the output information is used to indicate the contour information corresponding to the target object information; and filtering the output information based on the mask image to obtain the first image information.

[0060] Optionally, the shape of the object is controlled by Canny ControlNet. First, a line drawing image (Canny Edge) corresponding to the target object information is obtained as a visual control condition. Then, the target visual control network is called to process the line drawing image to obtain output information. For example, the line drawing image and the object word can be input into Canny ControlNet to output the edge contour information of the object. Then, the output information is filtered based on the mask image to obtain the first image information. For example, the edge contour information of the object is dotted with the object mask to obtain the area that needs to be visually controlled.

[0061] It should be noted that, by processing the output information of the target visual control network based on the mask image, a mask operation is performed on the parts that do not need to be controlled in the output layer of ControlNet, so that it can follow the instructions of the prompt and is not affected by the visual control. The edge contour of the object is effectively controlled, and the background blur problem existing in the local visual control method in the prior art is solved.

[0062] In order to realize the generation of text to image, in the image generation method provided in the first embodiment of the present application, a target image corresponding to the text information is generated based on the first image information, the target object information, the attribute information corresponding to the target object information and the target loss function corresponding to the target text image model, including: inputting the first image information, the target object information and the attribute information corresponding to the target object information into the target network module; taking the mask image as the constraint condition of the target loss function, iteratively updating the target network module according to the target loss function until a preset number of iterations is reached to obtain the target image corresponding to the text information.

[0063] Optionally, the target network module is a U-net network of a target text image model, and the area that needs visual control, object words, and attribute words corresponding to the object words are input into the U-net network. The mask image is used as a constraint condition of the target loss function, and then the U-net network is iteratively updated according to the target loss function until a preset number of iterations is reached, and the image corresponding to the prompt can be obtained.

[0064] By iteratively updating the target network module according to the target loss function, the distance between the attention map of the object and its corresponding attribute is shortened under the dual constraints of visual control and object mask, thereby achieving attribute binding of objects in the visual control area.

[0065] In order to improve the accuracy of attribute binding while controlling visual conditions, in the image generation method provided in Example 1 of the present application, the target loss function includes a first loss function and a second loss function, the targets corresponding to the first loss function and the second loss function are different, and iteratively updating the target network module according to the target loss function includes: obtaining an attention map of the target object information and an attention map of the attribute information corresponding to the target object information; calculating the divergence distance between the attention map of the target object information and the attention map of the attribute information corresponding to the target object information according to the first loss function; determining the focus area of ​​the attention map of the target object information and the focus area of ​​the attention map of the attribute information corresponding to the target object information; calculating the similarity between the focus area of ​​the attention map of the target object information and the focus area of ​​the attention map of the attribute information corresponding to the target object information according to the second loss function; and iteratively updating the target network module according to the divergence distance and the similarity.

[0066] Optionally, the first loss function is a language-guided loss, and the second loss function is a mask-guided loss. These two loss functions promote the alignment of the attribute-object word attention map by adjusting the attention map. By iteratively denoising and updating the latent space variables of the noise at each time step in the U-net network denoising process, the target network module is iteratively updated according to the target loss function. First, the attention map of the target object information and the attention map of the attribute information corresponding to the target object information are obtained. For example, the attention map of the object word is obtained from ControlNet, and the attention map of the attribute word is obtained from the trained SD model (the target text map large model). Among them, the attention map in the SD model represents the correlation between the text word and the visual pixel. In addition to reflecting the correlation between the text word and the visual pixel, the attention map in ControlNet can also reflect the correlation between the visual control condition and the visual pixel.

[0067] Optionally, a divergence distance between an attention map of the target object information and an attention map of the attribute information corresponding to the target object information is calculated based on a first loss function, and the first loss function aims to achieve a preset distance for the divergence distance. For example, the first loss function is the KL (Kullback-Leibler) divergence distance between the attention map of the object word and the attention map of the attribute word. By making the function value of the first loss function smaller, the distribution of the attention map of the object word and the attention map of the attribute word are brought closer, so that the object can correctly express its attributes while satisfying visual control.

[0068] Optionally, the second loss function is used to restrict the focus areas of the attention map of the object and its corresponding attribute words to a large extent within the object mask area. Therefore, the focus area of ​​the attention map of the target object information and the focus area of ​​the attention map of the attribute information corresponding to the target object information are first determined, and then the similarity between the focus area of ​​the attention map of the target object information and the focus area of ​​the attention map of the attribute information corresponding to the target object information is calculated based on the second loss function. The second loss function is used to gradually increase the degree of attention of the object word and the attribute word to the same area in the step-by-step denoising.

[0069] In order to achieve attribute binding in accordance with visual control, in the image generation method provided in Example 1 of the present application, iteratively updating the target network module based on the divergence distance and the similarity includes: iteratively updating the target network module with the divergence distance reaching a preset distance as the goal, and with the similarity reaching a preset similarity as the goal.

[0070] Optionally, in the process of iteratively updating the target network module based on the divergence distance and the similarity, the target network module is iteratively updated with the goal of the divergence distance reaching a preset distance and the similarity reaching a preset similarity. For example, the U-net network is iteratively updated with the goal of making the KL divergence distance between the attention map of the object word and the attention map of the attribute word smaller and with the goal of making the object word and the attribute word pay more attention to the same area.

[0071] It should be noted that the first loss function and the second loss function ensure that the more focused areas (with higher attention weights) of attribute words and object words are consistent within the object mask area, while maintaining precise control of visual conditions, and achieving attribute binding between object words and corresponding attribute words.

[0072] In order to extract the object words and attribute words in the prompt, in the image generation method provided in the first embodiment of the present application, obtaining the target object information and the attribute information corresponding to the target object information in the text information through the target text image large model includes: receiving the text information input by the target object through the target text image large model; calling the natural language processing tool to perform syntactic analysis on the text information to obtain the target object information and the attribute information corresponding to the target object information.

[0073] Optionally, the target text graph model receives text information (i.e., prompt) input by the user (i.e., the target object), and calls a natural language processing tool (such as a dependency parser, etc.) to perform syntactic analysis on the prompt. The natural language processing tool can identify the object words and corresponding attribute words in the sentence to form a combination of <object word, attribute word>. For example, after syntactic analysis of "a yellow dog, a green skateboard, and huge rocks on the mountaintop", we can get<dog,yellow> ,<skateboard,green> Two pairs of objects and their attribute relationships.

[0074] By performing syntactic analysis on the text information, the object and the attributes to be bound to the object can be obtained, which provides a basis for attribute binding.

[0075] In an optional embodiment, it is possible to use Figure 3 The schematic diagram shown realizes text-to-image generation. Figure 3 is a schematic diagram of an optional text-to-image generation process according to the first embodiment of the present application, such as Figure 3As shown in the figure, the entire framework uses the trained SD model to realize text-to-image generation, and uses the visual control network to realize the control of various visual conditions. For example, the trained SD model receives the prompt input by the user (such as: a yellow dog, a green skateboard and a boulder on the top of the mountain), calls the natural language processing tool to perform syntactic analysis on the prompt, and obtains the object and the attributes to be bound to the object, that is, the object word and the attribute word (such as <dog, yellow>, <skateboard, green>). The natural language processing tool returns the object word and the attribute word to the trained SD model and the visual control network. The trained SD model generates an object mask according to the object word (such as Figure 3 The trained SD model’s text encoder can convert object words into semantic vectors and input them into the encoder of the U-net network. The U-net network combines the semantic vectors to generate Figure 3 The pure noise shown in the figure is gradually removed to generate image information latent variables, that is, the input of the U-net network is the pure noise latent variable (for example, the initial pure noise is generated by a random function) and a semantic vector, and the output is a denoised latent variable. The U-net network combines the semantic vector to iteratively remove the noise in the pure noise latent variable step by step, and gradually injects semantic information into the latent variable. At the same time, based on the target loss function (i.e., attribute matching loss) and the mask image (i.e., object mask), the distance between the attention map of the object and its corresponding attribute is shortened under the dual constraints of visual control and object mask until denoising is completed, and a semantic denoised latent variable is obtained, and then the denoised latent variable is input into the image decoder (such as Figure 3 By obtaining the line drawing image corresponding to the object word (such as Figure 3 As the visual control condition, the line drawing image and the object word are input into the visual control network, the edge contour information of the object is output, and then the output information is filtered based on the mask image (such as Figure 3 As shown in the figure, the edge contour information of the object is dot-multiplied with the object mask), the area that needs visual control is obtained and input into the decoder; in the decoder, the denoised image information latent variable is converted into an image according to the area that needs visual control, which effectively controls the edge contour of the object and eliminates the interference of the Canny image background area on the prompt control.

[0076] In an embodiment of the present application, target object information in text information and attribute information corresponding to the target object information are obtained through a target text image large model, wherein the target text image large model is obtained by training an initial text image large model based on the attribute information and position information of the object in the training set image; a mask image corresponding to the target object information is determined, and the output information of the target visual control network is processed based on the mask image to obtain first image information, wherein the target visual control network is used to control the object shape corresponding to the target object information, and the first image information is used to indicate the visual control area corresponding to the target object information; a target image corresponding to the text information is generated based on the first image information, the target object information, the attribute information corresponding to the target object information, and the target loss function corresponding to the target text image large model, wherein the target loss function is used to limit the attribute binding between the target object information and the attribute information corresponding to the target object information to the visual control area In this way, the output information of the target visual control network is processed based on the mask image, so as to realize the shape control of the object within the mask and avoid the interference of the visual control on the image generation outside the mask. Through the target loss function corresponding to the target Vincent graph large model, it is realized under the dual constraints of visual control and object mask, the distance between the attention map of the object and its corresponding attribute is shortened, the alignment of the attention map and the control of local visual conditions are taken into account, the instruction following ability of the Vincent graph large model is improved, and accurate background generation and attribute binding are realized. The purpose of realizing the attribute binding between the object and its corresponding attribute while maintaining the accurate control of the visual conditions is achieved, thereby realizing the technical effect of improving the effect of the generated image, and further solving the technical problem that the Vincent graph large model in the related art performs attribute binding on the generated image under the local visual control condition, and the image background is blurred and the attribute binding accuracy is low, resulting in poor generated image effect.

[0077] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0078] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0079] Example 2

[0080] According to an embodiment of the present application, a method for generating a target cultural graph large model is also provided, such as Figure 4 As shown, the method includes:

[0081] Step S401, obtaining a training set, wherein the training set includes a plurality of image-text pair samples;

[0082] Step S402, before the self-attention mechanism of the middle layer of the initial text-image large model, adding a position network module to obtain the text-image large model to be trained, wherein the position network module is used to inject attribute information and position information of objects in the image in the image-text pair sample;

[0083] Step S403, training the to-be-trained large model of the text graph according to the training set to obtain a target large model of the text graph.

[0084] Optionally, the initial text-based image model uses the text-based image model Stable Diffusion (SD for short). Based on the training-free solution, although low-cost object position control and attribute binding can be achieved, since the attention map needs to be modified during the forward denoising process, this behavior is not aligned with the SD model training process, which also causes problems such as unnatural images and insufficient aesthetics to a certain extent, and is slightly insufficient for application in real scenes. Therefore, the image-text pair samples and the position information of significant objects in the image can be collected to construct a training set for training object generation under layout control.

[0085] For example, a position network module (position net) is added before the self-attention mechanism (self-attention) of the middle block layer of the U-net network of the SD model to inject the attributes and position information of the objects in the image to obtain the large model of the image to be trained. During training, other network parameters are frozen, and only the newly added positionnet module is fine-tuned to obtain the target large model of the image. This makes the trained large model of the image (i.e., the target large model of the image) have better control over the position and attribute binding of objects, and the resulting image is more natural and beautiful, and can be more adaptable to complex scenes.

[0086] Example 3

[0087] According to an embodiment of the present application, a method for generating an image is also provided. Figure 5 As shown, the method includes:

[0088] Step S501, obtaining text information uploaded by the client;

[0089] Step S502, obtaining target object information in the text information and attribute information corresponding to the target object information through the target Wensheng graph large model in the cloud server, wherein the target Wensheng graph large model is obtained by training the initial Wensheng graph large model according to the attribute information and position information of the object in the training set image; determining a mask image corresponding to the target object information, and processing the output information of the target visual control network based on the mask image to obtain first image information, wherein the target visual control network is used to control the object shape corresponding to the target object information, and the first image information is used to indicate the visual control area corresponding to the target object information; generating a target image corresponding to the text information based on the first image information, the target object information, the attribute information corresponding to the target object information, and the target loss function corresponding to the target Wensheng graph large model, wherein the target loss function is used to limit the attribute binding between the target object information and the attribute information corresponding to the target object information within the visual control area;

[0090] Step S503: Feedback the target image to the client.

[0091] Through the above scheme, the output information of the target visual control network is processed based on the mask image, so as to realize the shape control of the object within the mask and avoid the interference of the visual control on the image generation outside the mask. Through the target loss function corresponding to the target Vincent graph large model, it is realized under the dual constraints of visual control and object mask, the distance between the attention map of the object and its corresponding attribute is shortened, the alignment of the attention map and the control of local visual conditions are taken into account, the instruction following ability of the Vincent graph large model is improved, and accurate background generation and attribute binding are realized. The purpose of achieving the attribute binding between the object and its corresponding attribute while maintaining the accurate control of the visual conditions is achieved, thereby achieving the technical effect of improving the effect of the generated image, and further solving the technical problem that the Vincent graph large model in the related art performs attribute binding on the generated image under the local visual control condition, and the image background is blurred and the attribute binding accuracy is low, resulting in poor generated image effect.

[0092] In the cloud server, the specific method for generating the image is the same as that in the first embodiment, and will not be described again here.

[0093] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0094] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0095] Example 4

[0096] According to an embodiment of the present application, an image generating device for implementing the above-mentioned image generating method is also provided. Figure 6 As shown, the device includes: a first acquisition unit 601, a first processing unit 602, and a second processing unit 603.

[0097] A first acquisition unit 601 is used to acquire target object information and attribute information corresponding to the target object information in the text information through a target text image large model, wherein the target text image large model is obtained by training an initial text image large model according to the attribute information and position information of the object in the training set image;

[0098] A first processing unit 602 is used to determine a mask image corresponding to the target object information, and process output information of the target visual control network based on the mask image to obtain first image information, wherein the target visual control network is used to control the object shape corresponding to the target object information, and the first image information is used to indicate a visual control area corresponding to the target object information;

[0099] The second processing unit 603 is used to generate a target image corresponding to the text information based on the first image information, the target object information, the attribute information corresponding to the target object information, and the target loss function corresponding to the target text image model, wherein the target loss function is used to limit the attribute binding between the target object information and the attribute information corresponding to the target object information within the visual control area.

[0100] In the image generation device provided in the fourth embodiment of the present application, the target object information and the attribute information corresponding to the target object information in the text information are acquired by the first acquisition unit 601 through the target Wensheng graph large model, wherein the target Wensheng graph large model is obtained by training the initial Wensheng graph large model according to the attribute information and position information of the object in the training set image; the first processing unit 602 determines the mask image corresponding to the target object information, and processes the output information of the target visual control network based on the mask image to obtain the first image information, wherein the target visual control network is used to control the object shape corresponding to the target object information, and the first image information is used to indicate the visual control area corresponding to the target object information; the second processing unit 603 generates the target image corresponding to the text information based on the first image information, the target object information, the attribute information corresponding to the target object information, and the target loss function corresponding to the target Wensheng graph large model, wherein the target loss function is used to limit the attribute binding between the target object information and the attribute information corresponding to the target object information within the visual control area.

[0101] In this scheme, the output information of the target visual control network is processed based on the mask image, so as to realize the shape control of the object within the mask and avoid the interference of visual control on the image generation outside the mask. Through the target loss function corresponding to the target Vincent graph large model, it is realized that under the dual constraints of visual control and object mask, the distance between the attention map of the object and its corresponding attribute is shortened, and the alignment of the attention map and the control of local visual conditions are taken into account, thereby improving the instruction following ability of the Vincent graph large model, realizing accurate background generation and attribute binding, and achieving the purpose of attribute binding between the object and its corresponding attribute while maintaining accurate control of visual conditions, thereby achieving the technical effect of improving the effect of the generated image, and further solving the technical problem that the Vincent graph large model in the related art performs attribute binding on the generated image under local visual control conditions, and the image background is blurred and the attribute binding accuracy is low, resulting in poor generated image effect.

[0102] Optionally, in the image generating device provided in Embodiment 4 of the present application, the first processing unit 602 includes: an acquisition subunit, used to acquire a line draft image corresponding to the target object information; a first processing subunit, used to call the target visual control network to process the line draft image to obtain output information of the target visual control network, wherein the output information is used to indicate contour information corresponding to the target object information; and a second processing subunit, used to perform information filtering processing on the output information based on the mask image to obtain first image information.

[0103] Optionally, in the image generating device provided in Embodiment 4 of the present application, the second processing unit 603 includes: a third processing sub-unit, used to input the first image information, target object information, and attribute information corresponding to the target object information into a target network module; a fourth processing sub-unit, used to use the mask image as a constraint condition of the target loss function, and iteratively update the target network module according to the target loss function until a preset number of iterations is reached to obtain a target image corresponding to the text information.

[0104] Optionally, in the image generation device provided in Example 4 of the present application, the target loss function includes a first loss function and a second loss function, the targets corresponding to the first loss function and the second loss function are different, and the fourth processing subunit includes: an acquisition module, used to acquire the attention map of the target object information and the attention map of the attribute information corresponding to the target object information; a first calculation module, used to calculate the divergence distance between the attention map of the target object information and the attention map of the attribute information corresponding to the target object information based on the first loss function; a determination module, used to determine the focus area of ​​the attention map of the target object information and the focus area of ​​the attention map of the attribute information corresponding to the target object information; a second calculation module, used to calculate the similarity between the focus area of ​​the attention map of the target object information and the focus area of ​​the attention map of the attribute information corresponding to the target object information based on the second loss function; a processing module, used to iteratively update the target network module based on the divergence distance and the similarity.

[0105] Optionally, in the image generating device provided in the fourth embodiment of the present application, the processing module includes: a processing submodule, which is used to iteratively update the target network module with the divergence distance reaching a preset distance as a goal and the similarity reaching a preset similarity as a goal.

[0106] Optionally, in the image generation device provided in Example 4 of the present application, the first acquisition unit 601 includes: a receiving subunit, used to receive text information input by the target object through the target text image model; an analysis subunit, used to call a natural language processing tool to perform syntactic analysis on the text information to obtain target object information and attribute information corresponding to the target object information.

[0107] It should be noted that the above-mentioned first acquisition unit 601, first processing unit 602, and second processing unit 603 correspond to steps S201 to S203 in Example 1, and the above-mentioned units and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.

[0108] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0109] Example 5

[0110] According to an embodiment of the present application, a device for generating a target culture graph large model for implementing the above-mentioned method for generating a target culture graph large model is also provided. Figure 7 As shown, the device includes: a second acquisition unit 701, a third processing unit 702, and a fourth processing unit 703.

[0111] A second acquisition unit 701 is used to acquire a training set, wherein the training set includes a plurality of image-text pair samples;

[0112] The third processing unit 702 is used to add a position network module before the self-attention mechanism of the middle layer of the initial text-image large model to obtain the text-image large model to be trained, wherein the position network module is used to inject attribute information and position information of objects in the image in the image-text pair sample;

[0113] The fourth processing unit 703 is used to train the Wensheng graph large model to be trained according to the training set to obtain a target Wensheng graph large model.

[0114] It should be noted here that the above-mentioned second acquisition unit 701, third processing unit 702, and fourth processing unit 703 correspond to steps S401 to S403 in Example 2, and the instances and application scenarios implemented by the above-mentioned units and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 2.

[0115] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 2 as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 2.

[0116] Example 6

[0117] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0118] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0119] In this embodiment, the electronic device can execute the program code of the following steps in the image generation method: obtaining target object information and attribute information corresponding to the target object information in the text information through the target Wensheng graph large model, wherein the target Wensheng graph large model is obtained by training the initial Wensheng graph large model based on the attribute information and position information of the object in the training set image; determining a mask image corresponding to the target object information, and processing the output information of the target visual control network based on the mask image to obtain first image information, wherein the target visual control network is used to control the object shape corresponding to the target object information, and the first image information is used to indicate the visual control area corresponding to the target object information; generating a target image corresponding to the text information based on the first image information, the target object information, the attribute information corresponding to the target object information, and the target loss function corresponding to the target Wensheng graph large model, wherein the target loss function is used to limit the attribute binding between the target object information and the attribute information corresponding to the target object information within the visual control area.

[0120] The above-mentioned electronic device can also execute the program code of the following steps in the image generation method: obtaining a line draft image corresponding to the target object information; calling the target visual control network to process the line draft image to obtain output information of the target visual control network, wherein the output information is used to indicate contour information corresponding to the target object information; performing information filtering processing on the output information based on the mask image to obtain the first image information.

[0121] The above-mentioned electronic device can also execute the program code of the following steps in the image generation method: inputting the first image information, the target object information and the attribute information corresponding to the target object information into the target network module; using the mask image as the constraint condition of the target loss function, iteratively updating the target network module according to the target loss function until a preset number of iterations is reached to obtain the target image corresponding to the text information.

[0122] The above-mentioned electronic device can also execute the program code of the following steps in the image generation method: obtaining an attention map of the target object information and an attention map of the attribute information corresponding to the target object information; calculating the divergence distance between the attention map of the target object information and the attention map of the attribute information corresponding to the target object information based on a first loss function; determining the focus area of ​​the attention map of the target object information and the focus area of ​​the attention map of the attribute information corresponding to the target object information; calculating the similarity between the focus area of ​​the attention map of the target object information and the focus area of ​​the attention map of the attribute information corresponding to the target object information based on a second loss function; and iteratively updating the target network module based on the divergence distance and the similarity.

[0123] The electronic device can also execute the program code of the following steps in the image generation method: iteratively updating the target network module with the divergence distance reaching a preset distance as the goal and the similarity reaching a preset similarity as the goal.

[0124] The electronic device can also execute the program code of the following steps in the image generation method: receiving text information input by the target object through the target text image model; calling the natural language processing tool to perform syntactic analysis on the text information to obtain the target object information and the attribute information corresponding to the target object information.

[0125] Optionally, Figure 8 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 8 As shown, the electronic device 80 may include: one or more ( Figure 8 Only one is shown in the figure) processor 802, memory 804. The electronic device 80 may also include a storage controller, through which the memory 804 is controlled and managed; the electronic device 80 may also include a peripheral interface, through which the radio frequency module, audio module and display screen are connected.

[0126] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the image generation method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned image generation method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0127] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the target object information in the text information and the attribute information corresponding to the target object information through the target Wensheng graph large model, wherein the target Wensheng graph large model is obtained by training the initial Wensheng graph large model based on the attribute information and position information of the object in the training set image; determine the mask image corresponding to the target object information, and process the output information of the target visual control network based on the mask image to obtain the first image information, wherein the target visual control network is used to control the object shape corresponding to the target object information, and the first image information is used to indicate the visual control area corresponding to the target object information; generate the target image corresponding to the text information based on the first image information, the target object information, the attribute information corresponding to the target object information and the target loss function corresponding to the target Wensheng graph large model, wherein the target loss function is used to limit the attribute binding between the target object information and the attribute information corresponding to the target object information within the visual control area.

[0128] Optionally, the processor may also execute the program code of the following steps: obtaining a line drawing image corresponding to the target object information; calling the target visual control network to process the line drawing image to obtain output information of the target visual control network, wherein the output information is used to indicate contour information corresponding to the target object information; performing information filtering processing on the output information based on the mask image to obtain first image information.

[0129] Optionally, the processor may also execute the following program code steps: inputting the first image information, target object information, and attribute information corresponding to the target object information into the target network module; using the mask image as a constraint condition of the target loss function, iteratively updating the target network module according to the target loss function until a preset number of iterations is reached to obtain a target image corresponding to the text information.

[0130] Optionally, the processor may also execute program code for the following steps: obtaining an attention map of the target object information and an attention map of the attribute information corresponding to the target object information; calculating a divergence distance between the attention map of the target object information and the attention map of the attribute information corresponding to the target object information based on a first loss function; determining a focus area of ​​the attention map of the target object information and a focus area of ​​the attention map of the attribute information corresponding to the target object information; calculating a similarity between a focus area of ​​the attention map of the target object information and a focus area of ​​the attention map of the attribute information corresponding to the target object information based on a second loss function; and iteratively updating the target network module based on the divergence distance and the similarity.

[0131] Optionally, the processor may also execute the program code of the following steps: iteratively updating the target network module with the divergence distance reaching a preset distance as a goal, and with the similarity reaching a preset similarity as a goal.

[0132] Optionally, the processor may also execute the following program code: receiving text information input by the target object through the target text-image model; calling a natural language processing tool to perform syntactic analysis on the text information to obtain target object information and attribute information corresponding to the target object information.

[0133] It can be understood by those skilled in the art that Figure 8 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Figure 8 The structure of the electronic device is not limited. Figure 8 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 8 Different configurations shown.

[0134] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0135] Example 7

[0136] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image generation method provided in the first embodiment.

[0137] Optionally, in this embodiment, the above storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.

[0138] Example 8

[0139] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and when the computer program is executed by a processor, the image generation method provided in the first embodiment is implemented.

[0140] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0141] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0142] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0143] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0145] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0146] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image generation method, characterized in that: include: Obtaining target object information in the text information and attribute information corresponding to the target object information through a target text image large model, wherein the target text image large model is obtained by training an initial text image large model based on the attribute information and position information of the object in the training set image; Determine a mask image corresponding to the target object information, and process output information of a target visual control network based on the mask image to obtain first image information, wherein the target visual control network is used to control the object shape corresponding to the target object information, and the first image information is used to indicate a visual control area corresponding to the target object information; Generate a target image corresponding to the text information based on the first image information, the target object information, the attribute information corresponding to the target object information, and the target loss function corresponding to the target text image model, wherein the target loss function is used to limit the attribute binding between the target object information and the attribute information corresponding to the target object information within the visual control area; The input of the target visual control network is a line drawing image corresponding to the target object information, the output information is used to indicate the contour information corresponding to the target object information, and the mask image is used to perform information filtering processing on the output information; Wherein, generating a target image corresponding to the text information based on the first image information, the target object information, the attribute information corresponding to the target object information, and the target loss function corresponding to the target text image model includes: Inputting the first image information, the target object information and the attribute information corresponding to the target object information into a target network module, wherein the target network module is a U-net network of the target Wensheng graph large model; The mask image is used as a constraint condition of the target loss function, and the target network module is iteratively updated according to the target loss function to obtain a target image corresponding to the text information.

2. The method according to claim 1, characterized in that The output information of the target visual control network is processed based on the mask image to obtain the first image information, including: Obtaining a line drawing image corresponding to the target object information; Calling the target visual control network to process the line drawing image to obtain output information of the target visual control network; The output information is subjected to information filtering processing based on the mask image to obtain the first image information.

3. The method according to claim 1, characterized in that The target loss function includes a first loss function and a second loss function, the first loss function and the second loss function correspond to different targets, and iteratively updating the target network module according to the target loss function includes: Obtaining an attention map of the target object information and an attention map of attribute information corresponding to the target object information; Calculating a divergence distance between an attention map of the target object information and an attention map of attribute information corresponding to the target object information according to the first loss function; Determine a focus area of ​​the attention map of the target object information and a focus area of ​​the attention map of the attribute information corresponding to the target object information; Calculating the similarity between the attention area of ​​the attention map of the target object information and the attention area of ​​the attention map of the attribute information corresponding to the target object information according to the second loss function; The target network module is iteratively updated according to the divergence distance and the similarity.

4. The method according to claim 3, characterized in that Iteratively updating the target network module according to the divergence distance and the similarity includes: The target network module is iteratively updated with the divergence distance reaching a preset distance as a goal and the similarity reaching a preset similarity as a goal.

5. The method according to claim 1, characterized in that Acquiring target object information in text information and attribute information corresponding to the target object information through the target text image model includes: Receiving the text information input by the target object through the target text map model; A natural language processing tool is called to perform syntax analysis on the text information to obtain the target object information and attribute information corresponding to the target object information.

6. A method for generating a large model of a target cultural graph, characterized in that: include: Acquire a training set, wherein the training set includes a plurality of image-text pair samples; Before the self-attention mechanism of the middle layer of the initial text-image large model, a position network module is added to obtain the text-image large model to be trained, wherein the position network module is used to inject attribute information and position information of objects in the image in the image-text pair sample; The large model of the Vincent graph to be trained is trained according to the training set to obtain the target large model of the Vincent graph as described in any one of claims 1 to 5.

7. An image generation method, characterized in that: include: Get the text information uploaded by the client; In the cloud server, target object information in the text information and attribute information corresponding to the target object information are obtained through a target Wensheng graph large model, wherein the target Wensheng graph large model is obtained by training an initial Wensheng graph large model based on the attribute information and position information of the object in the training set image; a mask image corresponding to the target object information is determined, and output information of a target visual control network is processed based on the mask image to obtain first image information, wherein the target visual control network is used to control the object shape corresponding to the target object information, and the first image information is used to indicate a visual control area corresponding to the target object information; a target image corresponding to the text information is generated based on the first image information, the target object information, the attribute information corresponding to the target object information, and a target loss function corresponding to the target Wensheng graph large model, wherein the target loss function is used to limit the attribute binding between the target object information and the attribute information corresponding to the target object information within the visual control area; Feeding back the target image to the client; The input of the target visual control network is a line drawing image corresponding to the target object information, the output information is used to indicate the contour information corresponding to the target object information, and the mask image is used to perform information filtering processing on the output information; Wherein, generating a target image corresponding to the text information based on the first image information, the target object information, the attribute information corresponding to the target object information, and the target loss function corresponding to the target text image model includes: Inputting the first image information, the target object information and the attribute information corresponding to the target object information into a target network module, wherein the target network module is a U-net network of the target Wensheng graph large model; The mask image is used as a constraint condition of the target loss function, and the target network module is iteratively updated according to the target loss function to obtain a target image corresponding to the text information.

8. An image generating device, characterized in that: include: A first acquisition unit is used to acquire target object information in text information and attribute information corresponding to the target object information through a target text image large model, wherein the target text image large model is obtained by training an initial text image large model based on the attribute information and position information of the object in the training set image; A first processing unit is used to determine a mask image corresponding to the target object information, and to process output information of a target visual control network based on the mask image to obtain first image information, wherein the target visual control network is used to control an object shape corresponding to the target object information, and the first image information is used to indicate a visual control area corresponding to the target object information; a second processing unit, configured to generate a target image corresponding to the text information based on the first image information, the target object information, the attribute information corresponding to the target object information, and a target loss function corresponding to the target text image model, wherein the target loss function is configured to limit the attribute binding between the target object information and the attribute information corresponding to the target object information within the visual control area; The input of the target visual control network is a line drawing image corresponding to the target object information, the output information is used to indicate the contour information corresponding to the target object information, and the mask image is used to perform information filtering processing on the output information; Wherein, the second processing unit includes: A third processing subunit is used to input the first image information, the target object information and the attribute information corresponding to the target object information into a target network module, wherein the target network module is a U-net network of the target Wensheng graph large model; The fourth processing subunit is used to use the mask image as a constraint condition of the target loss function, iteratively update the target network module according to the target loss function, and obtain a target image corresponding to the text information.

9. A device for generating a large model of a target cultural graph, characterized in that: include: A second acquisition unit is used to acquire a training set, wherein the training set includes a plurality of image-text pair samples; A third processing unit is used to add a position network module before the self-attention mechanism of the middle layer of the initial Wensheng graph large model to obtain a Wensheng graph large model to be trained, wherein the position network module is used to inject attribute information and position information of objects in the image in the image-text pair sample; The fourth processing unit is used to train the large model of the Wensheng graph to be trained according to the training set to obtain the target large model of the Wensheng graph according to any one of claims 1 to 5.

10. An electronic device, characterized in that: include: A memory storing an executable program; A processor, used to run the program, wherein the program executes the image generation method of any one of claims 1 to 5 and 7 when running.

11. A computer program product, characterized in that The method comprises a computer program or instructions, which, when executed by a processor, implements the image generation method according to any one of claims 1 to 5 and 7.

Images