Image generation method, apparatus, device, and medium

Abstract

The application relates to an image generation method, device, equipment and medium, and more particularly to the technical field of artificial intelligence. The method comprises the following steps: acquiring an initial scene description text; predicting scene layout information corresponding to the scene description text according to a text semantic feature of the scene description text; the scene layout information is used for representing a relative position relationship between scene objects in a scene described by the scene description text; adding the scene layout information in the scene description text to obtain a target scene description text; acquiring an initial noise image, and performing noise reduction on the noise image according to a text semantic feature of the target scene description text to obtain a target image; the target image is used for presenting a scene; a position relationship between scene objects in the presented scene meets the relative position relationship. The method can improve the image generation accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an image generation method and device, equipment and a medium. BACKGROUND

[0002] With the development of artificial intelligence technology, text-to-image technology has emerged. Text-to-image refers to generating an image of a scene described by text based on the input text. With the development of science and technology, text-to-image technology has been widely used in people's daily life. For example, image generation is performed on each sentence text in a script that needs to be shot, and the generated image can be provided to the director as a reference before shooting. For another example, image generation is performed on a scene that needs to be drawn, and the generated image can be provided to the painter for reference to inspire the painter to make deeper creation.

[0003] In the traditional technology, an open source text-to-image model is usually directly used to generate an image corresponding to the text, and the generated image often does not match the scene described by the input text, resulting in low image generation accuracy. SUMMARY

[0004] Therefore, it is necessary to provide an image generation method, device, equipment and medium capable of improving image generation accuracy to solve the above technical problems.

[0005] In a first aspect, the present application provides an image generation method, which comprises:

[0006] obtaining an initial scene description text;

[0007] predicting scene layout information corresponding to the scene description text according to a text semantic feature of the scene description text; the scene layout information is used to represent a relative position relationship between scene objects in a scene described by the scene description text;

[0008] adding the scene layout information in the scene description text to obtain a target scene description text;

[0009] obtaining an initial noise image, and performing noise reduction on the noise image according to a text semantic feature of the target scene description text to obtain a target image;

[0010] The target image is used to present the scene, and the position relationship between the scene objects in the presented scene meets the relative position relationship.

[0011] In a second aspect, the present application provides an image generation device, which comprises:

[0012] an obtaining module configured to obtain an initial scene description text;

[0013] a prediction module configured to predict scene layout information corresponding to the scene description text according to a text semantic feature of the scene description text, wherein the scene layout information is used to represent relative positional relationships between scene objects in a scene described by the scene description text;

[0014] an adding module configured to add the scene layout information into the scene description text to obtain a target scene description text;

[0015] a noise reduction module configured to obtain an initial noise image, and perform noise reduction on the noise image according to a text semantic feature of the target scene description text to obtain a target image, wherein the target image is used to present the scene, and a positional relationship between scene objects in the presented scene satisfies the relative positional relationships.

[0016] In an embodiment, the scene layout information comprises a scene layout category, and the prediction module is further configured to perform layout classification processing on the scene description text according to a text semantic feature of the scene description text to obtain a scene layout category to which the scene description text belongs.

[0017] In an embodiment, the apparatus further comprises:

[0018] a determination module configured to determine style information corresponding to the scene description text;

[0019] The adding module is further configured to add the scene layout information and the style information into the scene description text to obtain a target scene description text.

[0020] In an embodiment, the style information comprises a style category, the determination module is further configured to determine a creative text to which the scene description text belongs, the creative text comprises a plurality of scene description texts for describing a scene, a text semantic feature of the creative text is extracted according to scene description texts in the creative text, a style classification processing is performed on the creative text according to a text semantic feature of the creative text to obtain a style category to which the creative text belongs, and the style category to which the creative text belongs is taken as the style category corresponding to the scene description text.

[0021] In an embodiment, the noise reduction module is further configured to obtain a first image feature of the noise image, perform convolution processing on the text semantic feature of the target scene description text and the first image feature to obtain a second image feature, and perform decoding on the second image feature to obtain the target image after noise reduction on the noise image.

[0022] In an embodiment, the noise reduction module is further configured to diffuse sample first image features of the noise image at a first encoding time to obtain first image features of the noise image at a second encoding time; the first encoding time is located before the second encoding time; take the first image features of the noise image at the second encoding time as the reference image features to be reduced in the current round, take the second encoding time as the current encoding time, and perform convolution processing on the text semantic features of the target scene description text and the reference image features in the current round to obtain the intermediate state image features after noise reduction corresponding to the current encoding time in the current round; take the previous encoding time of the current encoding time as the current encoding time of the next round, take the next round as the current round, take the intermediate state image features as the reference image features to be reduced in the current round, and perform iterative convolution processing until the current encoding time is the first encoding time, and stop iterative noise reduction, and take the intermediate state image features corresponding to the first encoding time as the second image features.

[0023] In an embodiment, the scene layout information is obtained by a trained layout recognition model; the device further comprises:

[0024] The first training module is configured to obtain at least one first sample scene description text; the first sample scene description text is labeled with reference scene layout information; perform layout prediction on the first sample scene description text by a to-be-trained layout recognition model to obtain predicted scene layout information; perform iterative training on the to-be-trained layout recognition model according to the difference between the predicted scene layout information and the reference scene layout information, and obtain the trained layout recognition model.

[0025] In an embodiment, the target image is obtained by a trained image generation model; the device further comprises:

[0026] The second training module is configured to: obtain at least one sample image-text pair, wherein the sample image-text pair comprises a second sample scene description text and a reference image, the second sample scene description text comprises scene layout information corresponding to the second sample scene description text, and the reference image is used to present a scene described by the second sample scene description text; for each sample image-text pair, add noise to the reference image in the sample image-text pair to obtain a sample noise image, and input the sample noise image and the second sample scene description text in the sample image-text pair into a to-be-trained image generation model, so as to obtain a predicted image corresponding to the sample image-text pair by performing noise reduction on the sample noise image based on the second sample scene description text through the to-be-trained image generation model; and perform iterative training on the to-be-trained image generation model according to differences between the predicted image corresponding to each sample image-text pair and a corresponding reference image, so as to obtain a trained image generation model.

[0027] In one embodiment, the second sample scene description text further comprises style information corresponding to the second sample scene description text, the sample image-text pair used in each iteration in the iterative training process comprises at least one batch of sample image-text pairs, the reference image in each batch of sample image-text pairs comprises a reference layout image and a reference style image, and the second training module is further configured to, for each batch in each iteration of the training, train the to-be-trained image generation model according to differences between the predicted image corresponding to each sample image-text pair in the batch and a corresponding reference image, so as to obtain an image generation model corresponding to the batch, and when an iteration stop condition is met, take the image generation model corresponding to the last batch in the last iteration of the training as the trained image generation model.

[0028] In one embodiment, the reference image in each batch further comprises a sample style transfer image, the sample style transfer image is an image obtained by performing style transfer processing on the reference layout image, and the sample style transfer image has a layout corresponding to the reference layout image and a style used in the style transfer processing.

[0029] In one embodiment, the reference layout image in each batch of sample image-text pairs is an image extracted from a pre-generated layout image group, images in the same layout image group have the same layout, and the device further comprises:

[0030] The first generation module is configured to: obtain at least one video; perform foreground erasing on each image in each video to obtain a background image; and perform clustering on the background images according to image layout features corresponding to the background images, so as to obtain at least one layout image group.

[0031] In an embodiment, the first generation module is further configured to, for each image in each of the videos, perform saliency identification on the image to obtain a saliency image of the image, wherein a gray value of each pixel point in the saliency image represents a saliency degree of the pixel point, generate a mask image according to the gray values of the pixel points in the saliency image, and the mask image represents a salient region in the image, and erase the salient region in the image according to the mask image to obtain a background image corresponding to the image.

[0032] In an embodiment, the reference style image in each batch of sample image-text pairs is an image extracted from a pre-generated group of style images, and each image in the same group of style images has the same style. The device further comprises:

[0033] A second generation module configured to obtain at least one video, wherein each of the videos corresponds to a style, sample at least one video image from each video, and combine the sampled video images to obtain a group of style images corresponding to the video.

[0034] In an embodiment, the second training module is further configured to, for each sample image-text pair, determine a first loss value according to a difference between a predicted image feature of the predicted image corresponding to the sample image-text pair and a reference image feature of the corresponding reference image, determine a second loss value according to a difference between a pixel value of a pixel point in the predicted image corresponding to the sample image-text pair and a pixel value of a pixel point in the corresponding reference image, fuse the first loss value and the second loss value to obtain a target loss value corresponding to the sample image-text pair, and iteratively train the image generation model to be trained according to the target loss values corresponding to the sample image-text pairs to obtain a trained image generation model.

[0035] In a third aspect, a computer device is provided, which includes a memory and a processor, and the memory stores a computer program, and the processor implements the steps in the method embodiments of the present application when executing the computer program.

[0036] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program, when executed by a processor, implements the steps in the method embodiments of the present application.

[0037] In a fifth aspect, a computer program product is provided, which includes a computer program, and the computer program, when executed by a processor, implements the steps in the method embodiments of the present application.

[0038] The image generation method, device, equipment, medium and computer program product described above, by obtaining an initial scene description text, and according to the text semantic features of the scene description text, predicting the scene layout information corresponding to the scene description text. The scene layout information is used to represent the relative position relationship between the scene objects in the scene described by the scene description text. The scene layout information is added in the scene description text to obtain a target scene description text with scene layout information. An initial noise image is obtained, and the noise image is denoised according to the text semantic features of the target scene description text with scene layout information. Since the text semantic features of the target scene description text contain the layout information of the scene, the generated target image can be used to present the scene described by the initial scene description text, and the position relationship between the scene objects in the presented scene meets the relative position relationship, thereby improving the image generation accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 An application environment diagram of an image generation method in an embodiment;

[0040] Figure 2 A flowchart of an image generation method in an embodiment;

[0041] Figure 3 A traditional image corresponding to a text generated by using an open-source text-to-image model;

[0042] Figure 4 A target image generated by the image generation method of the present application in an embodiment;

[0043] Figure 5 A model structure diagram used for image generation of the present application in an embodiment;

[0044] Figure 6 A sample data and training process diagram for a layout recognition model to be trained in an embodiment;

[0045] Figure 7 A style transfer process diagram in an embodiment;

[0046] Figure 8 A sample data composition diagram for training an image generation model to be trained in an embodiment;

[0047] Figure 9 A label composition diagram of sample data for training an image generation model to be trained in an embodiment;

[0048] Figure 10 An image in a video in an embodiment;

[0049] Figure 11This is a schematic diagram of a mask image in one embodiment;

[0050] Figure 12 This is a schematic diagram illustrating the sample data generation process designed for the layout recognition model and the image generation model to be trained in one embodiment.

[0051] Figure 13 This is a schematic diagram illustrating the image generation for each sentence in a script in one embodiment;

[0052] Figure 14 This is a flowchart illustrating the image generation method in another embodiment;

[0053] Figure 15 This is a structural block diagram of an image generation device in one embodiment;

[0054] Figure 16 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] The image generation method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on other servers. Terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be a single physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, cloud security, host security and other network security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms. Terminal 102 and server 104 can be directly or indirectly connected via wired or wireless communication; this application does not impose any restrictions.

[0057] The server 104 can obtain an initial scene description text, and predict scene layout information corresponding to the scene description text according to a text semantic feature of the scene description text. The scene layout information is used to represent a relative position relationship between scene objects in a scene described by the scene description text. The server 104 can add the scene layout information to the scene description text to obtain a target scene description text. The server 104 can obtain an initial noise image, and denoises the noise image according to a text semantic feature of the target scene description text to obtain a target image. The target image is used to present a scene, and a position relationship between scene objects in the presented scene meets the relative position relationship.

[0058] It can be understood that the server 104 can send the generated target image to the terminal 102 for display. It can be understood that the server 104 can also store the generated target image or perform corresponding image processing. This embodiment is not limited thereto, and it can be understood that Figure 1 The application scenarios in the above embodiments are only illustrative and are not limited thereto.

[0059] It should be noted that the image generation method in some embodiments of the present application uses artificial intelligence technology. For example, the text semantic feature of the scene description text and the text semantic feature of the target scene description text in the present application are features obtained by using artificial intelligence technology, and the scene layout information corresponding to the scene description text in the present application is information predicted by using artificial intelligence technology. In addition, the target image in the present application is also an image generated by using artificial intelligence technology. In order to facilitate understanding of artificial intelligence, the concept of artificial intelligence is described as follows. Specifically, artificial intelligence is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer, perceiving an environment, obtaining knowledge and using the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

[0060] In one embodiment, as shown in Figure 2 An image generation method is provided, which can be applied to a computer device, which can be a terminal or a server, and can be executed by the terminal or the server itself alone or through interaction between the terminal and the server. This embodiment takes the method applied to a computer device as an example for description, which includes the following steps.

[0061] In step 202, an initial scene description text is obtained.

[0062] The scene description text is a text for describing a scene. The scene is a scene expression composed of at least one scene object. The scene object is an object in the scene. For better understanding of the scene description text, an example is given. If the scene description text is "Zhang San and Li Si come to the inn at the foot of the mountain", the scene description text can include four scene objects, namely, Zhang San, Li Si, the foot of the mountain and the inn, and a specific scene expression is composed of the four scene objects.

[0063] In an embodiment, the computer device can obtain a creative text. It can be understood that the creative text includes a plurality of scene description texts for describing a scene. The computer device can disassemble the creative text to obtain a plurality of initial scene description texts.

[0064] In an embodiment, the creative text can specifically include at least one of a script, a novel, a poem or a prose.

[0065] In an embodiment, the computer device can disassemble the creative text at a sentence level, and take the disassembled sentence text as the scene description text. The computer device can also disassemble the creative text at a comma sentence level, and take the disassembled comma sentence as the scene description text. It can be understood that if the disassembled text is not satisfactory, the disassembled text can be further disassembled by manual operation. Specifically, the computer device can respond to the text disassembly operation to further disassemble the disassembled text to obtain the scene description text.

[0066] In an embodiment, the computer device can also obtain at least one short sentence text independent of each other, and directly take the obtained short sentence text as the scene description text. It can be understood that the short sentence text independent of each other means that there is no context relationship between the short sentence texts. The short sentence text refers to a text with a sentence length less than a preset length threshold.

[0067] In step 204, scene layout information corresponding to the scene description text is predicted according to a text semantic feature of the scene description text. The scene layout information is used to represent a relative position relationship between scene objects in the scene described by the scene description text.

[0068] For better understanding of the scene layout information, an example is given. If the scene description text is "Zhang San and Li Si come to the inn at the foot of the mountain", the scene description text can include four scene objects, namely, Zhang San, Li Si, the foot of the mountain and the inn, and the scene layout information corresponding to the scene description text can be used to represent a relative position relationship between the four scene objects. For example, the inn is located at the foot of the mountain, and Zhang San and Li Si are located in front of the inn.

[0069] In an embodiment, the computer device can encode features of the scene description text to obtain text semantic features of the scene description text. Further, the computer device can predict layout information of the scene description text according to the text semantic features of the scene description text to obtain scene layout information corresponding to the scene description text.

[0070] Step 206, adding scene layout information in the scene description text to obtain a target scene description text.

[0071] The target scene description text is a scene description text obtained after adding scene layout information in an initial scene description text. It can be understood that the initial scene description text is the scene description text before adding the scene layout information.

[0072] Specifically, the computer device can add scene layout information corresponding to the scene description text in the initial scene description text to obtain the target scene description text. It can be understood that the target scene description text adds scene layout information corresponding to the scene description text compared to the initial scene description text.

[0073] For example, if the initial scene description text is “Zhang San and Li Si two people come to the inn at the foot of the mountain”, the scene layout information corresponding to the initial scene description text is “layout 1”, and the target scene description text obtained by adding scene layout information in the scene description text can be “<layout 1> Zhang San and Li Si two people come to the inn at the foot of the mountain”. It can be understood that “layout 1” defines the relative position relationship between the four scene objects in the scene description text, namely Zhang San, Li Si, the foot of the mountain and the inn.

[0074] Step 208, obtaining an initial noise image, and denoising the noise image according to the text semantic features of the target scene description text to obtain a target image; wherein the target image is used to present a scene; the position relationship between the scene objects in the presented scene meets the relative position relationship.

[0075] The noise image is an image containing noise. The target image is an image obtained by denoising the noise image. It can be understood that the target image has less noise compared to the noise image.

[0076] Specifically, the computer device can generate an initial noise image based on random noise, and encode features of a target scene description text to obtain text semantic features of the target scene description text. Further, the computer device can denoise the initial noise image according to the text semantic features of the target scene description text to obtain a denoised target image. It can be understood that, since the computer device refers to the text semantic features of the target scene description text in the process of denoising the noise image, the denoised target image can be used to present the scene described by the corresponding scene description text, and the position relationship between scene objects in the presented scene meets the relative position relationship between scene objects in the scene described by the corresponding scene description text. The corresponding scene description text refers to an initial scene description text corresponding to the target scene description text.

[0077] For example, if the initial scene description text is "Zhang San and Li Si come to the inn at the foot of the mountain", the scene layout information corresponding to the initial scene description text is "layout 1", and the target scene description text is "<layout 1> Zhang San and Li Si come to the inn at the foot of the mountain", the denoised target image can be used to present the scene described by "Zhang San and Li Si come to the inn at the foot of the mountain", and the position relationship between the four scene objects of Zhang San, Li Si, the foot of the mountain and the inn meets the relative position relationship between the four scene objects of Zhang San, Li Si, the foot of the mountain and the inn defined by "layout 1".

[0078] In one embodiment, the computer device can encode features of the initial noise image to obtain image features of the noise image. Further, the computer device can denoise the image features of the noise image based on the text semantic features of the target scene description text to obtain denoised image features. The computer device can generate a denoised target image according to the denoised image features.

[0079] In one embodiment, the computer device can obtain a trained image generation model, and input the noise image and the target scene description text into the trained image generation model to denoise the noise image according to the text semantic features of the target scene description text through the trained image generation model to obtain the target image.

[0080] In the image generation method, an initial scene description text is obtained, and scene layout information corresponding to the scene description text is predicted according to a text semantic feature of the scene description text. The scene layout information is used to represent a relative position relationship between scene objects in a scene described by the scene description text. The scene layout information is added to the scene description text to obtain a target scene description text with the scene layout information. An initial noise image is obtained, and the noise image is denoised according to a text semantic feature of the target scene description text with the scene layout information. Since the text semantic feature of the target scene description text includes the scene layout information, the generated target image can be used to present the scene described by the initial scene description text, and the position relationship between the scene objects in the presented scene meets the relative position relationship, thereby improving the image generation accuracy.

[0081] A conventional method of directly generating a text corresponding image by using an open-source text-to-image model often generates an image that does not match a scene described by input text, thereby resulting in low image generation accuracy. Figure 3 As shown in FIG. 1, the input text is “Two people, namely Zhang San and Li Si, come to an inn at the foot of a mountain”, and the output image focuses more on the two people, thereby generating a scene image in which the two people face each other. The generated image does not match the scene described by the input text, thereby resulting in low image generation accuracy.

[0082] However, the present application can improve the image generation accuracy by adding scene layout information to a scene description text to obtain a target scene description text with the scene layout information, and guiding generation of a target image based on the target scene description text. Figure 4 As shown in FIG. 2, the input text is “Two people, namely Zhang San and Li Si, come to an inn at the foot of a mountain”, and the output image focuses more on the position relationship between the four scene objects, namely Zhang San, Li Si, the foot of the mountain, and the inn. The generated image matches the scene described by the input scene description text, thereby improving the image generation accuracy.

[0083] In one embodiment, the scene layout information includes a scene layout category; and the scene layout information corresponding to the scene description text is predicted according to a text semantic feature of the scene description text, including: performing layout classification processing on the scene description text according to the text semantic feature of the scene description text to obtain a scene layout category to which the scene description text belongs.

[0084] The scene layout category is a layout category corresponding to a distribution of scene objects in a scene described by the scene description text. It can be understood that different relative position relationships between the scene objects in the scene described by the scene description text can correspond to different layout categories.

[0085] Specifically, the computer device can predefine a plurality of scene layout categories, and for each initial scene description text, the computer device can calculate the probability that the scene description text belongs to each of the pre-defined scene layout categories according to the text semantic features of the scene description text, and take the scene layout category corresponding to the maximum probability as the scene layout category to which the scene description text belongs.

[0086] For example, the pre-defined plurality of scene layout categories can include layout 1, layout 2, layout 3, …, and layout n, where n is any positive integer. For the initial scene description text A, the computer device can calculate the probability that the scene description text A belongs to each of the pre-defined scene layout categories (i.e., layout 1, layout 2, layout 3, …, and layout n) according to the text semantic features of the scene description text A, and take the scene layout category corresponding to the maximum probability as the scene layout category to which the scene description text belongs. For example, if the calculated probability corresponding to the maximum probability is layout 3, then layout 3 is taken as the scene layout category to which the scene description text belongs.

[0087] In one embodiment, the computer device can obtain a trained layout recognition model, and input the initial scene description text into the trained layout recognition model to perform layout classification processing on the scene description text according to the text semantic features of the scene description text by the trained layout recognition model, to obtain the scene layout category to which the scene description text belongs.

[0088] In the above embodiment, the scene description text is intelligently classified and processed according to the text semantic features of the scene description text, which can accurately obtain the scene layout category to which the scene description text belongs, thereby obtaining accurate scene layout information and improving the accuracy of obtaining scene layout information.

[0089] In one embodiment, the method further includes determining style information corresponding to the scene description text, and adding the scene layout information in the scene description text to obtain a target scene description text, including adding the scene layout information and the style information in the scene description text to obtain the target scene description text.

[0090] The style information corresponding to the scene description text is used to represent that the scene described by the scene description text is presented by a corresponding style. The style can include at least one of an animation style, a Chinese animation style, an ink style, a modern live-action style, or a historical live-action style.

[0091] Specifically, the computer device can obtain an initial scene description text, and determine style information corresponding to the scene description text. The computer device can encode the initial scene description text to obtain text semantic features of the scene description text, and predict scene layout information corresponding to the scene description text according to the text semantic features of the scene description text. The computer device can add the scene layout information and the style information in the scene description text to obtain a target scene description text. It can be understood that the target scene description text includes the scene layout information and the style information. The computer device can obtain an initial noise image, and encode the target scene description text to obtain text semantic features of the target scene description text. Further, the computer device can denoise the noise image according to the text semantic features of the target scene description text to obtain a target image. It can be understood that the target image can be used to present a scene described by the initial scene description text, and each scene object in the presented scene satisfies a layout represented by the scene layout information, and the target image satisfies a style represented by the style information.

[0092] In one embodiment, the style information corresponding to the scene description text can be customized by a user. It can be understood that the user wants to generate a target image of a certain style, and sets the style information of the scene description text according to the desired style. The computer device can obtain the style information input by the user in response to a style setting operation of the user, and take the style information input by the user as the style information corresponding to the scene description text.

[0093] In one embodiment, the computer device can determine a creative text to which the scene description text belongs, and disassemble the creative text at a sentence level, and take a sentence text disassembled as the scene description text. Further, the computer device can obtain the style information input by the user in response to a style setting operation of the user for the creative text, and take the style information input by the user for the creative text as the style information corresponding to the scene description text. It can be understood that the style information of the creative text can be directly taken as the style information corresponding to each scene description text in the creative text.

[0094] In one embodiment, the computer device can also obtain at least one short sentence text independent of each other, and take the obtained short sentence text as the scene description text. Further, the computer device can obtain the style information input by the user in response to a style setting operation of the user for the short sentence text, and take the style information input by the user for the short sentence text as the style information corresponding to the scene description text.

[0095] In the above embodiment, by adding the scene layout information and the style information in the scene description text, the obtained target scene description text can include the scene layout information and the style information corresponding to the scene description text, so that the target image obtained by subsequently performing noise reduction on the noise image according to the text semantic features of the target scene description text can not only present the scene described by the initial scene description text and present the layout of each scene object in the presented scene satisfying the layout represented by the scene layout information, but also make the target image satisfy the style represented by the style information.

[0096] In one embodiment, the style information includes a style category; determining the style information corresponding to the scene description text includes: determining a creative text to which the scene description text belongs; the creative text includes a plurality of scene description texts created for describing a scene; extracting text semantic features of the creative text according to the scene description texts in the creative text; performing style classification processing on the creative text according to the text semantic features of the creative text to obtain a style category to which the creative text belongs, and taking the style category to which the creative text belongs as the style category corresponding to the scene description text.

[0097] Specifically, the style category is a category of the style presented by the image. The creative text includes a plurality of scene description texts created for describing a scene. The computer device can determine a creative text to which the scene description text belongs, and extract text semantic features of the creative text according to the scene description texts in the creative text. Furthermore, the computer device can perform style classification processing on the creative text according to the text semantic features of the creative text to obtain a style category to which the creative text belongs, and take the style category to which the creative text belongs as the style category corresponding to the scene description text. It can be understood that the style categories corresponding to the scene description texts in the same creative text are the same, i.e., the style category to which the creative text belongs.

[0098] In the above embodiment, the overall creative text is intelligently classified by the text semantic features of the creative text. Since there is a context relationship between each scene description text in the overall creative text, the style category to which the overall creative text belongs can be accurately obtained, so that the style category to which the overall creative text belongs is directly taken as the style category corresponding to the scene description text, which can improve the accuracy of determining the style category to which the scene description text belongs.

[0099] In one embodiment, the noise image is denoised according to the text semantic features of the target scene description text to obtain a target image, including: obtaining a first image feature of the noise image; performing convolution processing on the text semantic features of the target scene description text and the first image feature to obtain a second image feature; and decoding the second image feature to obtain the target image after noise reduction on the noise image.

[0100] The first image feature is an image feature of the initial noisy image, and the second image feature is an image feature obtained by convolving the text semantic feature of the target scene description text and the first image feature. It can be understood that the process of convolving the text semantic feature of the target scene description text and the first image feature is a process of denoising the initial noisy image. That is, the second image feature has less noise than the first image feature.

[0101] Specifically, the computer device can encode the initial noisy image to obtain the first image feature of the noisy image. The computer device can encode the target scene description text to obtain the text semantic feature of the target scene description text, and convolve the text semantic feature of the target scene description text and the first image feature to obtain the second image feature after convolution. Further, the computer device can decode the second image feature to obtain the target image after denoising the noisy image. It can be understood that the target image has less noise than the initial noisy image.

[0102] In one embodiment, the computer device can convolve the text semantic feature of the target scene description text and the first image feature multiple times to obtain the second image feature. It can be understood that the computer device can denoise the initial noisy image multiple times, and the image obtained after multiple denoising is taken as the target image.

[0103] In one embodiment, the computer device directly convolves the text semantic feature of the target scene description text and the first image feature once to obtain the second image feature. It can be understood that the computer device can denoise the initial noisy image once, and the image obtained after one-time denoising is taken as the target image.

[0104] In the above embodiments, by convolving the text semantic feature of the target scene description text and the first image feature of the noisy image, the second image feature after denoising can be obtained, and then the target image is generated based on the second image feature after denoising, which can improve the denoising effect of the noisy image.

[0105] In an embodiment, the text semantic feature of the target scene description text and the first image feature are convoluted to obtain a second image feature, including: diffusing and sampling the first image feature of the noise image at a first encoding time to obtain the first image feature of the noise image at a second encoding time; the first encoding time is located before the second encoding time; taking the first image feature of the noise image at the second encoding time as a reference image feature to be denoised in a current round, taking the second encoding time as a current encoding time, and convoluting the text semantic feature of the target scene description text and the reference image feature in the current round to obtain an intermediate state image feature denoised at the current encoding time corresponding to the current round; taking a previous encoding time of the current encoding time as a current encoding time of a next round, taking the next round as the current round, taking the intermediate state image feature as the reference image feature to be denoised in the current round, and iteratively convoluting to stop iterative denoising when the current encoding time is the first encoding time, and taking the intermediate state image feature corresponding to the first encoding time as the second image feature.

[0106] In an embodiment, the text semantic feature of the target scene description text and the first image feature are convoluted to obtain a second image feature, including: diffusing and sampling the first image feature of the noise image at a first encoding time to obtain the first image feature of the noise image at a second encoding time; the first encoding time is located before the second encoding time; taking the first image feature of the noise image at the second encoding time as a reference image feature to be denoised in a current round, taking the second encoding time as a current encoding time, and convoluting the text semantic feature of the target scene description text and the reference image feature in the current round to obtain an intermediate state image feature denoised at the current encoding time corresponding to the current round; taking a previous encoding time of the current encoding time as a current encoding time of a next round, taking the next round as the current round, taking the intermediate state image feature as the reference image feature to be denoised in the current round, and iteratively convoluting to stop iterative denoising when the current encoding time is the first encoding time, and taking the intermediate state image feature corresponding to the first encoding time as the second image feature.

[0107] Specifically, the computer device can diffuse sample the first image feature of the noisy image at the first encoding moment to obtain the first image feature of the noisy image at the second encoding moment. The computer device can perform multiple convolutional processing (i.e., multiple rounds of denoising processing) on the text semantic feature of the target scene description text and the first image feature to obtain the second image feature. It can be understood that the computer device can take the first image feature of the noisy image at the second encoding moment as the reference image feature to be denoised in the current round, take the second encoding moment as the current encoding moment, and perform convolutional processing on the text semantic feature of the target scene description text and the reference image feature in the current round to obtain the intermediate state image feature after denoising corresponding to the current encoding moment in the current round. The computer device can take the previous encoding moment of the current encoding moment as the current encoding moment of the next round, take the next round as the current round, take the intermediate state image feature as the reference image feature to be denoised in the current round, and return to perform the step of performing convolutional processing on the text semantic feature of the target scene description text and the reference image feature in the current round and subsequent steps to perform iterative convolutional processing until the current encoding moment is the first encoding moment. Stop iterative denoising, and take the intermediate state image feature corresponding to the first encoding moment as the second image feature. Further, the computer device can decode the second image feature to obtain the target image after denoising of the noisy image.

[0108] In one embodiment, as Figure 5As shown, the computer device can input the initial scene description text into the trained layout recognition model to perform feature coding on the initial scene description text by the trained layout recognition model, obtain the text semantic features of the scene description text, and perform layout recognition on the scene description text based on the text semantic features of the scene description text to obtain the scene layout information corresponding to the scene description text. Then, the computer device can add the scene layout information to the initial scene description text to obtain the target scene description text. The computer device can input the target scene description text into the trained image generation model to perform feature coding on the target scene description text by the trained image generation model to obtain the text semantic features of the target scene description text. The computer device can obtain the noise image and perform diffusion sampling on the first image features of the noise image at the first coding time (for example, the 0th coding time) to obtain the first image features of the noise image at the second coding time (for example, the Tth coding time, where T is any positive integer). The computer device can take the first image features of the noise image at the second coding time as the reference image features to be denoised in the current round, take the second coding time as the current coding time, perform convolution processing on the text semantic features of the target scene description text and the reference image features in the current round by the cross-attention network of the denoising unit in the trained image generation model to obtain the intermediate state image features after denoising corresponding to the current coding time in the current round. The computer device can take the previous coding time (for example, the T-1th time) of the current coding time as the current coding time in the next round, take the next round as the current round, take the intermediate state image features as the reference image features to be denoised in the current round, and return to perform the step of convolution processing on the text semantic features of the target scene description text and the reference image features in the current round and the subsequent steps to perform iterative convolution processing until the current coding time is the first coding time (i.e., the 0th coding time), and stop iterative denoising, take the intermediate state image features corresponding to the first coding time as the second image features, and decode the second image features to obtain the target image after denoising of the noise image.

[0109] In the above embodiment, in the coding space, the first image features of the noise image at the first coding time are diffused and sampled to obtain the first image features of the noise image at the second coding time. Since the first coding time is before the second coding time, each coding time between the first coding time and the second coding time can be sequentially denoised. Through multiple denoising of the noise image, a target image with lower noise level can be obtained, so that the image generation accuracy can be further improved.

[0110] In one embodiment, the scene layout information is determined by a trained layout recognition model; the method further includes: acquiring at least one first sample scene description text; the first sample scene description text is labeled with reference scene layout information; the layout of the first sample scene description text is predicted by the layout recognition model to be trained to obtain predicted scene layout information; and the layout recognition model to be trained is iteratively trained according to the difference between the predicted scene layout information and the reference scene layout information to obtain a trained layout recognition model.

[0111] The first sample scene description text is used to train the layout recognition model. The reference scene layout information is the actual scene layout information corresponding to the first sample scene description text, serving as a reference during the training process to guide the updating of model parameters. The predicted scene layout information is the scene layout information predicted by the layout recognition model during training.

[0112] Specifically, such as Figure 6 As shown, the computer device can acquire at least one first sample scene description text, wherein the first sample scene description text is annotated with reference scene layout information. The computer device can input the first sample scene description text into a layout recognition model to be trained (including an encoding layer and a classification layer) to predict the layout of the first sample scene description text through the layout recognition model to be trained, thereby obtaining predicted scene layout information. Furthermore, the computer device can determine the loss value for training the layout recognition model to be trained based on the difference between the predicted scene layout information and the reference scene layout information, and iteratively train the layout recognition model to be trained according to the determined loss value until the layout training stopping condition is met, thereby obtaining a trained layout recognition model.

[0113] In one embodiment, the stopping condition for layout training iteration can be that the number of iterations meets a preset number of layout training iterations, or that the loss value of training the layout recognition model to be trained is less than a preset layout training loss value.

[0114] In one embodiment, the loss value for training the layout recognition model can be determined by the following formula:

[0115]

[0116] Where M represents the number of layout categories represented by the reference scene layout information, p ic The first sample scene description text represents the predicted probability that sample i belongs to category c. ic Indicates whether the label of sample i is c (if it is c, then y icis 1, otherwise 0), and L represents a loss value for training the layout recognition model to be trained.

[0117] In the above embodiment, since the first sample scene description text is labeled with the reference scene layout information provided as a reference in the training process, training the layout recognition model to be trained by the first sample scene description text can improve the layout recognition accuracy of the trained layout recognition model.

[0118] In one embodiment, the target image is generated by the trained image generation model; the method further comprises: obtaining at least one sample image-text pair; the sample image-text pair comprises a second sample scene description text and a reference image; the second sample scene description text is added with scene layout information corresponding to the second sample scene description text; the reference image is used to present the scene described in the second sample scene description text; for each sample image-text pair, noise is added to the reference image in the sample image-text pair to obtain a sample noise image, and the sample noise image and the second sample scene description text in the sample image-text pair are input into the image generation model to be trained to perform noise reduction on the sample noise image based on the second sample scene description text by the image generation model to be trained, thereby obtaining a predicted image corresponding to the sample image-text pair; and the image generation model to be trained is iteratively trained according to the difference between the predicted image corresponding to each sample image-text pair and the corresponding reference image, thereby obtaining the trained image generation model.

[0119] In the above embodiment, since the first sample scene description text is labeled with the reference scene layout information provided as a reference in the training process, training the layout recognition model to be trained by the first sample scene description text can improve the layout recognition accuracy of the trained layout recognition model.

[0120] Specifically, the computer device can obtain at least one sample image-text pair, wherein the sample image-text pair comprises a second sample scene description text and a reference image. The second sample scene description text is added with scene layout information corresponding to the second sample scene description text. The scene layout information corresponding to the second sample scene description text is used to represent the relative positional relationship between scene objects in the scene described by the second sample scene description text. The reference image is used to present the scene described by the second sample scene description text. It can be understood that the positional relationship between scene objects in the presented scene satisfies the relative positional relationship between scene objects in the scene described by the second sample scene description text. Before training the image generation model to be trained, for each sample image-text pair, the computer device can add noise to the reference image in the sample image-text pair to obtain a sample noise image used in the training process. During the training of the image generation model to be trained, the computer device can input the sample noise image and the second sample scene description text in the sample image-text pair into the image generation model to be trained, so as to denoise the sample noise image based on the second sample scene description text by the image generation model to be trained, and obtain a predicted image corresponding to the sample image-text pair. Further, the computer device can determine a target loss value according to the difference between the predicted image corresponding to each sample image-text pair and the corresponding reference image, and iteratively train the image generation model to be trained according to the target loss value until a generation training iteration stop condition is met, and obtain a trained image generation model.

[0121] In one embodiment, the generation training iteration stop condition can be that the number of iterations meets a preset generation training iteration number, or the loss value of the training of the image generation model to be trained is less than a preset generation training loss value.

[0122] In the above embodiment, since the sample image-text pair comprises the second sample scene description text and the reference image provided for the training process as a reference, the second sample scene description text is added with the scene layout information corresponding to the second sample scene description text, and the reference image is used to present the scene described by the second sample scene description text, the training of the image generation model to be trained by the sample image-text pair can improve the image generation accuracy of the trained image generation model.

[0123] In an embodiment, the second sample scene description text further comprises style information corresponding to the second sample scene description text; the sample image-text pairs used in each iteration of the iterative training process comprise at least one batch of sample image-text pairs; the reference images in each batch of sample image-text pairs comprise a reference layout image and a reference style image; the trained image generation model is obtained by iteratively training the image generation model according to the difference between the predicted images and the corresponding reference images of each sample image-text pair, comprising: for each batch in each iteration of the training, the image generation model is trained according to the difference between the predicted images and the corresponding reference images of each sample image-text pair in the batch, and the image generation model corresponding to the batch is obtained; and when the iteration stopping condition is met, the image generation model corresponding to the last batch in the last iteration of the training is taken as the trained image generation model.

[0124] The reference layout image has a target layout, and the target layout is a layout represented by the scene layout information in the second sample scene description text corresponding to the reference layout image. The reference style image has a target style, and the target style is a style represented by the style information in the second sample scene description text corresponding to the reference style image.

[0125] Specifically, in each iteration of the training process, since the memory resources of the training machine, i.e., the computer device, are limited, the full amount of sample image-text pairs cannot be input into the image generation model to be trained at one time, and therefore, the full amount of sample image-text pairs need to be input into the image generation model to be trained in batches, and the reference images in each batch of sample image-text pairs comprise a reference layout image and a reference style image. When all batches or a preset number of batches are completed, it is considered that one iteration of the training is completed. For each batch in each iteration of the training, the computer device can determine a target loss value according to the difference between the predicted images and the corresponding reference images of each sample image-text pair in the batch, and train the image generation model to be trained according to the target loss value, to obtain the image generation model corresponding to the batch. When the iteration stopping condition is met, the computer device can take the image generation model corresponding to the last batch in the last iteration of the training as the trained image generation model.

[0126] In the above embodiment, since the reference images in each batch of sample image-text pairs comprise a reference layout image and a reference style image, the layout and the style can be simultaneously focused on when the image generation model to be trained is adjusted each time, and the problem of overfitting of the image generation model to be trained in the training process can be avoided, and the image generation accuracy of the trained image generation model is further improved.

[0127] In an embodiment, the reference image in each batch further comprises a sample style transfer image; the sample style transfer image is an image obtained by performing a migration process on the reference layout image for a different style; the sample style transfer image has the layout corresponding to the reference layout image and the style used in the migration process.

[0128] In an embodiment, as shown in Figure 7 the computer device can input the reference layout image and the reference style image into the trained style transfer model to perform a migration process on the reference layout image for the style represented by the reference style image through the style transfer model, and obtain a sample style transfer image having both the layout corresponding to the reference layout image and the style corresponding to the reference style image.

[0129] For example, the reference layout image comprises a reference layout image 1 satisfying layout 1 and a reference layout image 2 satisfying layout 2, and the reference style image comprises a reference style image 1 satisfying style 1. The computer device can perform a migration process on the reference layout image 1 of layout 1 for style 1 to obtain a sample style transfer image 1. It can be understood that the sample style transfer image 1 has layout 1 and style 1 used in the migration process. The computer device can perform a migration process on the reference layout image 2 of layout 2 for style 1 to obtain a sample style transfer image 2. It can be understood that the sample style transfer image 2 has layout 2 and style 1 used in the migration process.

[0130] In an embodiment, as shown in Figure 8 the reference image in each batch of sample image-text pairs comprises a reference layout image, a reference style image and a sample style transfer image. For each batch, the sample image-text pairs in the batch are input into the image generation model to be trained, and the image generation model to be trained is iteratively trained according to the target loss value corresponding to each sample image-text pair to obtain the trained image generation model.

[0131] The computer device can obtain an initial image generation model, which can be understood as an image generation model that has been initially trained through an open source data set. The computer device can further train the initial image generation model through fine-tuning. Specifically, the reference image in each batch of sample image-text pairs comprises a reference layout image, a reference style image and a sample style transfer image, which can be understood as Figure 9As shown, each fine-tuning training of the initial image generation model can simultaneously include layout labels, style labels, and labels after layout and style combination, which can avoid bias of training sample data in the training process, thereby avoiding overfitting of the trained image generation model. It can be understood that fine-tuning of the initial image generation model only needs to update the parameters of the denoising unit in the initial image generation model, and the parameters of other units in the initial image generation model can not need to be adjusted.

[0132] In the above embodiment, in addition to the reference layout image and the reference style image, the reference image in each batch of sample image-text pairs also includes a sample style transfer image, which enhances the diversity of the reference image in the sample image-text pair, makes the training sample of the initial image generation model more abundant, and can better train the initial image generation model, and further improves the image generation accuracy of the trained image generation model.

[0133] In one embodiment, the reference layout image in each batch of sample image-text pairs is an image extracted from a pre-generated layout image group; the images in the same layout image group have the same layout; the method further comprises: obtaining at least one video; performing foreground erasing on each image in each video respectively to obtain a background image; clustering the background images according to the image layout features corresponding to each background image respectively to obtain at least one layout image group.

[0134] Specifically, the computer device can obtain at least one video, and perform foreground erasing on each image in each video respectively to obtain a background image. The computer device can respectively encode the features of each background image to obtain image layout features corresponding to each background image. Further, the computer device can cluster the background images according to the image layout features corresponding to each background image respectively to obtain at least one clustering cluster, and take each clustering cluster as a different layout image group.

[0135] In one embodiment, for each image in each video, the computer device can perform saliency recognition on the image to obtain a saliency region of the image. It can be understood that the saliency region can be regarded as the foreground of the image. The computer device can erase the saliency region of the image to obtain a background image corresponding to the image after foreground erasing.

[0136] In the above embodiment, by performing foreground erasing on each image in the video respectively, the background can be separated from the image to obtain a reference layout image, which avoids overfitting of the image background and the foreground, so that the image generation model trained by using the sample image-text pair containing the reference layout image has higher image generation accuracy.

[0137] In one embodiment, the foreground of each image in each video is erased respectively to obtain a background image, including: for each image in each video, performing saliency identification on the image to obtain a saliency image of the image; the gray value of each pixel point in the saliency image is used to represent the saliency degree of each pixel point; generating a mask image according to the gray value of each pixel point in the saliency image; the mask image is used to represent the salient region in the image; erasing the salient region in the image according to the mask image to obtain the background image corresponding to the image.

[0138] Specifically, for each image in each video, the computer device can determine the gray value of each pixel point in the image respectively to obtain a saliency image of the image, wherein the gray value of each pixel point in the saliency image is used to represent the saliency degree of each pixel point. The computer device can generate a mask image corresponding to the object according to the gray value of each pixel point in the saliency image, wherein the mask image is used to represent the salient region in the image. Further, the computer device can erase the salient region in the image according to the mask image to obtain the background image corresponding to the image.

[0139] In one embodiment, for each image in each video, the computer device can input the image into the trained saliency identification model to determine the gray value of each pixel point in the image respectively through the trained saliency identification model to obtain a saliency image of the image.

[0140] In one embodiment, for each image in each video, the computer device can compare the gray value of each pixel point in the saliency image with the preset gray threshold respectively, and set the gray value of the pixel point with the gray value less than the gray threshold to 0, and set the gray value of the pixel point with the gray value greater than or equal to the gray threshold to 1, to obtain the mask image corresponding to the object.

[0141] In one embodiment, for each image in each video, the computer device can take the gray value of each pixel point in the mask image of the image respectively, that is, (1-mask). For example, the gray value of the pixel point is 1 after taking the inverse, and the gray value of the pixel point is 0 after taking the inverse. The computer can multiply the image after taking the inverse of the gray value, that is, (1-mask) x image, to realize the erasing of the salient region in the image, to obtain the background image corresponding to the image.

[0142] In one embodiment, as Figure 10As shown, the images (a), (b), (c), (d), (e), (f) and (g) in the video are all images with backgrounds and foregrounds, for example, the foregrounds of the images (a), (b), (c), (d), (e) and (f) are animals, and the backgrounds are at least one of grasslands, slopes or forests. The foreground of the image (g) is a sofa, and the background is a room. As shown in the image (g), the foreground of the image (g) is a sofa, and the background is a room. Figure 11 As shown, the computer device can perform saliency recognition on the images respectively to obtain the saliency regions corresponding to the images respectively. It can be understood that the saliency regions can be regarded as the foregrounds of the images (i.e., the animals or the sofa, i.e., the white region part in the image (g)). Figure 11 The computer device can erase the saliency regions (white region part) of the images to obtain the background images of the images after erasing the foregrounds (i.e., at least one of the grasslands, slopes or forests).

[0143] In the above embodiment, by performing saliency recognition on the images, the saliency images of the images can be obtained, according to the gray values corresponding to the respective pixels in the saliency images, the mask images representing the saliency regions in the images can be generated, and then according to the mask images, the saliency regions in the images can be erased, which can improve the accuracy of erasing the saliency regions.

[0144] In one embodiment, the reference style image in each batch of sample image-text pairs is an image extracted from a pre-generated group of style images; the images in the same group of style images have the same style; the method further comprises: obtaining at least one video; wherein one video corresponds to one style; for each video, sampling at least one video image from the video, and combining the sampled video images to obtain a group of style images corresponding to the video.

[0145] Specifically, the computer device can obtain at least one video, and it can be understood that one video corresponds to one style. For each video, the computer device can sample at least one video image from the video, and combine the sampled video images to obtain a group of style images corresponding to the video.

[0146] In one embodiment, for each video, the computer device can perform feature extraction on the images in the video to obtain image features, and perform clustering on the images in the video according to the extracted image features to obtain at least one image set corresponding to the video. Wherein, the subject image contents of the images in the same image set are the same. The computer device can sample at least one image from each image set respectively, and combine the sampled images to obtain a group of style images corresponding to the video. Wherein, the subject image content is the object in the image as the subject. For example, two sheep running on a vast grassland, then "two sheep" and "grassland" belong to the subject image content.

[0147] In the above embodiment, since one video corresponds to one style, for each video, at least one video image sampled from the video forms a style image group corresponding to the video, which improves the generation accuracy of the style image group.

[0148] In one embodiment, the image generation model to be trained is iteratively trained according to the differences between the prediction images and the corresponding reference images of each sample text pair, to obtain the trained image generation model, including: for each sample text pair, determining a first loss value according to the difference between the prediction image features of the prediction image corresponding to the sample text pair and the reference image features of the corresponding reference image; determining a second loss value according to the difference between the pixel values of the pixel points in the prediction image corresponding to the sample text pair and the pixel values of the pixel points in the corresponding reference image; fusing the first loss value and the second loss value to obtain a target loss value corresponding to the sample text pair; iteratively training the image generation model to be trained according to the target loss values corresponding to each sample text pair to obtain the trained image generation model.

[0149] The first loss value is a loss value determined based on the difference between the prediction image features of the prediction image corresponding to the sample text pair and the reference image features of the corresponding reference image. The second loss value is a loss value determined based on the difference between the pixel values of the pixel points in the prediction image corresponding to the sample text pair and the pixel values of the pixel points in the corresponding reference image. The target loss value is a loss value obtained by fusing the first loss value and the second loss value. It can be understood that the target loss value simultaneously covers the first loss value and the second loss value.

[0150] Specifically, for each sample text pair, the computer device can determine a first loss value according to the difference between the prediction image features of the prediction image corresponding to the sample text pair and the reference image features of the corresponding reference image, and determine a second loss value according to the difference between the pixel values of the pixel points in the prediction image corresponding to the sample text pair and the pixel values of the pixel points in the corresponding reference image. The computer device can fuse the first loss value and the second loss value to obtain a target loss value corresponding to the sample text pair. Further, the computer device can iteratively train the image generation model to be trained according to the target loss values corresponding to each sample text pair until a generation iteration stop condition is met, to obtain the trained image generation model.

[0151] In one embodiment, the computer device can perform weighted processing on the first loss value and the second loss value to obtain the target loss value corresponding to the sample text pair.

[0152] In one embodiment, the first loss value can be calculated by the following formula:

[0153]

[0154] wherein, n1 represents the number of pixel points in the image, y i1 represents the reference image feature of the reference image, represents the predicted image feature of the predicted image, and Loss1 represents the first loss value.

[0155] In one embodiment, the second loss value can be calculated by the following formula:

[0156]

[0157] wherein, n2 represents the number of pixel points in the image, y i2 represents the pixel value of the pixel point in the reference image, represents the pixel value of the pixel point in the predicted image, and Loss2 represents the second loss value.

[0158] In the above embodiment, since the corresponding target loss value of the sample image-text pair can simultaneously cover the first loss value and the second loss value, the target loss value can further improve the image generation accuracy of the trained image generation model by iteratively training the image generation model to be trained.

[0159] In one embodiment, as Figure 12As shown, it is a generation process of training samples designed for the layout recognition model to be trained and the image generation model to be trained. Specifically, the computer device can obtain a plurality of videos (such as TV series 1, TV series 2, movie 1, movie 2 and movie 3 respectively corresponding to the videos), and perform frame extraction processing (such as one frame per second) on the plurality of videos respectively. The computer device can perform shot processing (the subject content in the same shot image is the same) on each image obtained by frame extraction, extract a shot representative image for each shot (such as extracting the image at the middle time as the shot representative of the shot), and perform image feature extraction on each shot representative image, perform style clustering on each shot representative image based on the extracted image features, and obtain at least one clustering cluster for the shot representative image. It can be understood that the styles of the images in the same cluster are the same. The computer device can perform style sampling for each clustering cluster for the shot representative image to obtain a style image group. It can be understood that at least one image is sampled in a cluster, and a style image group is constructed based on the sampled image, and the styles of the images in the style image group are the same. For example, the style image group can include a style 1 corresponding to a style image group 1, a style 2 corresponding to a style image group 2, a style 3 corresponding to a style image group 3, a style 4 corresponding to a style image group 4, and a style 5 corresponding to a style image group 5. It can be understood that each TV series or movie can correspond to a style. It can be understood that in each iteration of training the image generation model to be trained, the reference style image in each batch of sample image-text pairs is extracted from the style image groups.

[0160] With reference to Figure 12 , the computer device can perform foreground erasure on each shot representative image to obtain a plurality of image sets of background images (such as image set 1, image set 2, image set 3, image set 4 and image set 5), and it can be understood that each TV series or movie can correspond to an image set. The computer device can perform histogram equalization on each background image to adjust the pixel values of each pixel point in each background image, thereby adjusting the over-bright or over-dark areas in the background image and improving the quality of the background image. The computer device can perform feature extraction on the adjusted background image, and perform layout clustering based on the extracted image features to obtain at least one layout clustering cluster, and it can be understood that each image in the same layout clustering cluster has the same or similar layout. Since there can be images with large layout differences in the layout clustering cluster, the layout clustering cluster is filtered by manual cleaning to delete the layout clustering cluster corresponding to the images with large layout differences, and the cleaned layout image group is obtained. It can be understood that each image in the same layout image group has the same layout. It can be understood that in each iteration of training the image generation model to be trained, the reference layout image in each batch of sample image-text pairs is extracted from the layout image groups.

[0161] With reference to Figure 12 , the computer device can perform text conversion on each image in the layout clustering cluster to obtain description text corresponding to each image in the layout clustering cluster. For the same image, the layout corresponding to the image is taken as the annotation information of the description text of the image, and a first sample scene description text corresponding to the image is obtained. It can be understood that the first sample scene description text is annotated with reference scene layout information. It can be understood that the layout recognition model of the present application is trained by the first sample scene description text corresponding to each image in the layout clustering cluster.

[0162] In one embodiment, as Figure 13 indicated, the creative text of the present application includes a script, and the scene description text includes a sentence in the script describing a scene. A director needs to shoot a TV series, and can first generate images for each sentence in the script. The generated images can be provided to the director for reference before shooting the TV series. Specifically, the computer can obtain the script and perform sentence-level disassembly on the script to obtain multiple sentences (e.g., sentence 1-1, sentence 1-2, sentence 2, sentence 3-1, etc.), i.e., initial scene description text. The computer device can input the disassembled sentences into the trained layout recognition model for layout prediction to obtain scene layout information corresponding to each sentence, and add the scene layout information to the corresponding sentence to obtain target scene description text. Further, the computer device can input the target scene description text and the initial noise image into the trained image generation model to generate a target image through the trained image generation model and store it. Further, the director can preview the target image corresponding to each sentence in the script. It can be understood that the present application can further disassemble the split sentences multiple times to obtain sentences disassembled further. The image generation method of the further disassembled sentences is the same as that of the above-mentioned sentences. It can also be understood that after the director previews the target image, if the generated result is not satisfactory, at least one operation of deleting or adding each sentence in the script can be performed. After the deleting or adding operation is completed, image generation can be performed on each sentence in the script again.

[0163] As Figure 14 indicated, in one embodiment, an image generation method is provided, which can be applied to a computer device. The computer device can be a terminal or a server, and can be executed by the terminal or the server itself alone, or can be realized through interaction between the terminal and the server. This embodiment takes the method applied to the computer device as an example for description, and the method specifically includes the following steps:

[0164] Step 1402, obtaining at least one first sample scene description text; the first sample scene description text is annotated with reference scene layout information.

[0165] At step 1404, the first sample scene description text is subjected to layout prediction by the layout recognition model to be trained, to obtain predicted scene layout information.

[0166] At step 1406, the layout recognition model to be trained is subjected to iterative training according to a difference between the predicted scene layout information and the reference scene layout information, to obtain a trained layout recognition model.

[0167] At step 1408, at least one sample image-text pair is obtained; the sample image-text pair includes a second sample scene description text and a reference image; the second sample scene description text has added therein scene layout information and style information corresponding to the second sample scene description text; and the reference image is used to present a scene described by the second sample scene description text.

[0168] At step 1410, for each sample image-text pair, noise is added to the reference image in the sample image-text pair to obtain a sample noise image, and the sample noise image and the second sample scene description text in the sample image-text pair are input to the image generation model to be trained, so that the image generation model to be trained is used to denoise the sample noise image based on the second sample scene description text, to obtain a predicted image corresponding to the sample image-text pair.

[0169] At step 1412, the image generation model to be trained is subjected to iterative training according to a difference between the predicted image corresponding to each sample image-text pair and the corresponding reference image, to obtain a trained image generation model.

[0170] At step 1414, an initial scene description text is obtained, and style information corresponding to the scene description text is determined.

[0171] At step 1416, the scene description text is input to the trained layout recognition model, so that the trained layout recognition model is used to predict scene layout information corresponding to the scene description text according to text semantic features of the scene description text.

[0172] At step 1418, the scene layout information and the style information are added to the scene description text, to obtain a target scene description text.

[0173] At step 1420, an initial noise image is obtained, and the noise image and the target scene description text are input to the trained image generation model, so that the trained image generation model is used to obtain a first image feature of the noise image.

[0174] At step 1422, the first image feature of the noise image at a first encoding time is subjected to diffusion sampling, to obtain a first image feature of the noise image at a second encoding time; the first encoding time is located before the second encoding time.

[0175] Step 1424, taking the first image feature of the noise image at the second encoding moment as the reference image feature to be denoised in the current round, taking the second encoding moment as the current encoding moment, and performing convolution processing on the text semantic feature of the target scene description text and the reference image feature in the current round to obtain the intermediate state image feature after denoising corresponding to the current encoding moment in the current round.

[0176] Step 1426, taking the previous encoding moment of the current encoding moment as the current encoding moment of the next round, taking the next round as the current round, taking the intermediate state image feature as the reference image feature to be denoised in the current round, and performing iterative convolution processing until the current encoding moment is the first encoding moment, and stopping the iterative denoising, and taking the intermediate state image feature corresponding to the first encoding moment as the second image feature.

[0177] Step 1428, decoding the second image feature to obtain a target image after denoising of the noise image.

[0178] The application also provides an application scenario of the image generation method. Specifically, the image generation method can be applied to a shooting image generation scenario of a wuxia script. It can be understood that the creative text of the application includes a wuxia script, and the scene description text of the application includes a wuxia scene description text describing a wuxia scene in the wuxia script. In a specific application scenario, a film director needs to shoot a wuxia film, and can first generate images for each wuxia scene description text in the wuxia script. The generated images can be provided to the film director for reference before shooting the film. Through the picture reference before shooting, the shooting time of the wuxia film can be shortened, and the budget estimation can be quickly performed, which has an important positive impact on the film and television shooting industry. Specifically, the application can predict each wuxia scene description text through the trained layout recognition model, and realize image generation for the wuxia scene description text through the trained image generation model.

[0179] In the process of training the layout recognition model to be trained, the computer device can obtain at least one first sample wuxia scene description text. The first sample wuxia scene description text is labeled with reference scene layout information. The layout recognition model to be trained is used to predict the first sample wuxia scene description text to obtain predicted scene layout information. According to the difference between the predicted scene layout information and the reference scene layout information, the layout recognition model to be trained is iteratively trained to obtain the trained layout recognition model.

[0180] In the process of training the image generation model to be trained, the computer device can obtain at least one sample image-text pair; the sample image-text pair includes a second sample martial arts scene description text and a reference image; the second sample martial arts scene description text is added with scene layout information and style information corresponding to the second sample martial arts scene description text; the reference image is used to present the martial arts scene described in the second sample martial arts scene description text. For each sample image-text pair, noise is added to the reference image in the sample image-text pair to obtain a sample noise image, and the sample noise image and the second sample martial arts scene description text in the sample image-text pair are input into the image generation model to be trained, so as to obtain a predicted image corresponding to the sample image-text pair by the image generation model to be trained based on the second sample martial arts scene description text to denoise the sample noise image. According to the difference between the predicted image corresponding to each sample image-text pair and the corresponding reference image, the image generation model to be trained is iteratively trained to obtain a trained image generation model.

[0181] After the layout recognition model to be trained and the image generation model to be trained are trained, that is, in the application stage, the computer device can obtain a martial arts script, and perform sentence-level disassembly on the martial arts script to obtain a plurality of initial martial arts scene description texts. The computer device can determine the style information (such as any one of an animation style, a domestic animation style, an ink style, a modern real person style, and a historical real person style) corresponding to the martial arts scene description text. The martial arts scene description text is input into the trained layout recognition model to predict the scene layout information corresponding to the martial arts scene description text according to the text semantic features of the martial arts scene description text by the trained layout recognition model. The scene layout information and the style information are added in the martial arts scene description text to obtain a target martial arts scene description text. An initial noise image is obtained, and the noise image and the target martial arts scene description text are input into the trained image generation model to obtain a first image feature of the noise image by the trained image generation model. The first image feature of the noise image at a first encoding time is diffused and sampled to obtain a first image feature of the noise image at a second encoding time; the first encoding time is located before the second encoding time.

[0182] The computer device can use the first image feature of the noisy image at the second encoding time as the reference image feature to be denoised in the current round, and the second encoding time as the current encoding time. It then performs convolution processing on the textual semantic features of the target martial arts scene description text and the reference image feature of the current round to obtain the denoised intermediate image feature corresponding to the current encoding time. The previous encoding time is used as the current encoding time of the next round, and the next round is used as the current round. The intermediate image feature is used as the reference image feature to be denoised in the current round for iterative convolution processing until the current encoding time is the first encoding time, at which point iterative denoising stops, and the intermediate image feature corresponding to the first encoding time is used as the second image feature. The second image feature is decoded to obtain the denoised target image of the noisy image. It can be understood that the obtained target images are the images corresponding to the martial arts scene description texts in the martial arts script, and these target images can be provided to film directors for reference before filming. The image generation method of this application can improve the generation accuracy of martial arts images corresponding to martial arts scripts.

[0183] This application also provides another application scenario where the image generation method described above is applied. Specifically, this image generation method can be applied to the generation of reference images during the artist's creative process. It is understood that the image generation method of this application can generate an image of the scene to be painted, and the generated image can be provided to the artist for reference, inspiring deeper creative work. It is understood that the scene description text in this application can include painting scene description text. Specifically, the artist can input a painting scene description text for the scene to be created, and the computer device can use the image generation method of this application to generate an image based on the input painting scene description text, obtaining an image that conforms to the painting scene description text, allowing the artist to engage in deeper creative work and create more excellent paintings.

[0184] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially, these steps are not necessarily executed in that order. Unless otherwise expressly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the above embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same coding moment, but can be executed at different coding moments. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0185] In one embodiment, such as Figure 15 As shown, an image generation apparatus 1500 is provided, which specifically includes:

[0186] The acquisition module 1502 is configured to acquire an initial scene description text.

[0187] The prediction module 1504 is configured to predict scene layout information corresponding to the scene description text according to a text semantic feature of the scene description text, wherein the scene layout information is used to represent relative position relationships between scene objects in a scene described by the scene description text.

[0188] The adding module 1506 is configured to add the scene layout information into the scene description text to obtain a target scene description text.

[0189] The noise reduction module 1508 is configured to acquire an initial noise image, and perform noise reduction on the noise image according to a text semantic feature of the target scene description text to obtain a target image, wherein the target image is used to present a scene, and position relationships between scene objects in the presented scene satisfy the relative position relationships.

[0190] In an embodiment, the scene layout information includes a scene layout category, and the prediction module 1504 is further configured to perform layout classification processing on the scene description text according to the text semantic feature of the scene description text to obtain a scene layout category to which the scene description text belongs.

[0191] In an embodiment, the apparatus further includes:

[0192] The determination module is configured to determine style information corresponding to the scene description text.

[0193] The adding module 1506 is further configured to add the scene layout information and the style information into the scene description text to obtain the target scene description text.

[0194] In an embodiment, the style information includes a style category, and the determination module is further configured to determine a creative text to which the scene description text belongs, wherein the creative text includes a plurality of scene description texts for describing a scene, extract a text semantic feature of the creative text according to the scene description texts in the creative text, perform style classification processing on the creative text according to the text semantic feature of the creative text to obtain a style category to which the creative text belongs, and take the style category to which the creative text belongs as the style category corresponding to the scene description text.

[0195] In an embodiment, the noise reduction module 1508 is further configured to acquire a first image feature of the noise image, perform convolution processing on the text semantic feature of the target scene description text and the first image feature to obtain a second image feature, and decode the second image feature to obtain the target image after noise reduction on the noise image.

[0196] In an embodiment, the noise reduction module 1508 is further configured to diffuse sample the first image feature of the noisy image at a first encoding time to obtain a first image feature of the noisy image at a second encoding time; the first encoding time is located before the second encoding time; take the first image feature of the noisy image at the second encoding time as the reference image feature to be de-noised in the current round, take the second encoding time as the current encoding time, and perform convolution processing on the text semantic feature of the target scene description text and the reference image feature in the current round to obtain an intermediate state image feature after de-noising corresponding to the current encoding time in the current round; take the previous encoding time of the current encoding time as the current encoding time of the next round, take the next round as the current round, take the intermediate state image feature as the reference image feature to be de-noised in the current round, and perform iterative convolution processing until the current encoding time is the first encoding time, and stop iterative de-noising, and take the intermediate state image feature corresponding to the first encoding time as the second image feature.

[0197] In an embodiment, the scene layout information is obtained by a trained layout recognition model; the apparatus further comprises:

[0198] The first training module is configured to obtain at least one first sample scene description text; the first sample scene description text is labeled with reference scene layout information; perform layout prediction on the first sample scene description text by the to-be-trained layout recognition model to obtain predicted scene layout information; and perform iterative training on the to-be-trained layout recognition model according to the difference between the predicted scene layout information and the reference scene layout information to obtain the trained layout recognition model.

[0199] In an embodiment, the target image is obtained by a trained image generation model; the apparatus further comprises:

[0200] The second training module is configured to obtain at least one sample image-text pair; the sample image-text pair comprises a second sample scene description text and a reference image; the second sample scene description text is added with scene layout information corresponding to the second sample scene description text; the reference image is used to present the scene described by the second sample scene description text; for each sample image-text pair, add noise to the reference image in the sample image-text pair to obtain a sample noisy image, and input the sample noisy image and the second sample scene description text in the sample image-text pair into the to-be-trained image generation model, so as to obtain a predicted image corresponding to the sample image-text pair by the to-be-trained image generation model based on the second sample scene description text to de-noise the sample noisy image; and perform iterative training on the to-be-trained image generation model according to the difference between the predicted image corresponding to each sample image-text pair and the corresponding reference image to obtain the trained image generation model.

[0201] In an embodiment, the second sample scene description text further comprises style information corresponding to the second sample scene description text; the sample image-text pair used in each iteration of the iterative training process comprises at least one batch of sample image-text pairs; the reference image in each batch of sample image-text pairs comprises a reference layout image and a reference style image; the second training module is further configured to, for each batch in each iteration of the training, train the image generation model to be trained based on a difference between a predicted image corresponding to each sample image-text pair in the batch and a corresponding reference image, to obtain an image generation model corresponding to the batch; and when the iteration stopping condition is met, the image generation model corresponding to the last batch in the last iteration of the training is taken as the trained image generation model.

[0202] In an embodiment, the reference image in each batch further comprises a sample style transfer image; the sample style transfer image is an image obtained by performing style transfer processing on the reference layout image; the sample style transfer image has a layout corresponding to the reference layout image and a style used in the style transfer processing.

[0203] In an embodiment, the reference layout image in each batch of sample image-text pairs is an image extracted from a pre-generated layout image group; the images in the same layout image group have the same layout; and the device further comprises:

[0204] The first generation module is configured to obtain at least one video; perform foreground erasing on each image in each video to obtain a background image; and cluster the background images based on image layout features corresponding to the background images to obtain at least one layout image group.

[0205] In an embodiment, the first generation module is further configured to, for each image in each video, perform saliency identification on the image to obtain a saliency image of the image; the gray value of each pixel point in the saliency image is used to represent the saliency of the pixel point; generate a mask image based on the gray values of the pixel points in the saliency image; the mask image is used to represent a saliency region in the image; and perform erasing on the saliency region in the image based on the mask image to obtain a background image corresponding to the image.

[0206] In an embodiment, the reference style image in each batch of sample image-text pairs is an image extracted from a pre-generated style image group; the images in the same style image group have the same style; and the device further comprises:

[0207] The second generation module is configured to obtain at least one video; wherein one video corresponds to one style; for each video, sample at least one video image from the video, and combine the sampled video images to obtain a style image group corresponding to the video.

[0208] In an embodiment, the second training module is further configured to determine, for each sample-text-image pair, a first loss value according to a difference between a predicted image feature of a predicted image corresponding to the sample-text-image pair and a reference image feature of a corresponding reference image; determine a second loss value according to a difference between a pixel value of a pixel point in the predicted image corresponding to the sample-text-image pair and a pixel value of the pixel point in the corresponding reference image; fuse the first loss value and the second loss value to obtain a target loss value corresponding to the sample-text-image pair; and perform iterative training on the image generation model to be trained according to the target loss values respectively corresponding to the sample-text-image pairs, to obtain the trained image generation model.

[0209] The image generation apparatus described above, by obtaining an initial scene description text, and predicting scene layout information corresponding to the scene description text according to a text semantic feature of the scene description text. The scene layout information is used to represent the relative positional relationship between the scene objects in the scene described by the scene description text. The scene layout information is added in the scene description text to obtain a target scene description text with scene layout information. An initial noise image is obtained, and the noise image is denoised according to the text semantic feature of the target scene description text with scene layout information. Since the text semantic feature of the target scene description text contains the layout information of the scene, the generated target image can be used to present the scene described by the initial scene description text, and the positional relationship between the scene objects in the presented scene meets the relative positional relationship, thereby improving the image generation accuracy.

[0210] Each module in the image generation apparatus described above can be realized by software, hardware, and combinations thereof, in whole or in part. Each module described above can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be called and executed by a processor to perform the operations corresponding to each module.

[0211] In an embodiment, a computer device is provided, which can be a server or a terminal. The internal structure diagram of the computer device can be as shown in Figure 16As shown in the figure. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to realize an image generation method.

[0212] Those skilled in the art can understand that, Figure 16 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0213] In one embodiment, a computer device is also provided, including a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the steps in each of the above method embodiments.

[0214] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by the processor to realize the steps in each of the above method embodiments.

[0215] In one embodiment, a computer program product is provided, which includes a computer program, and the computer program is executed by the processor to realize the steps in each of the above method embodiments.

[0216] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0217] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0218] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0219] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. An image generation method characterized by, The method comprises: disassembling the creative text to obtain a plurality of initial scene description texts; predicting scene layout information corresponding to the scene description text according to text semantic features of the scene description text; the scene layout information is used to represent relative position relationships between scene objects in a scene described by the scene description text; classifying the creative text according to text semantic features of the creative text to obtain a style category to which the creative text belongs, and taking the style category to which the creative text belongs as a style category of the initial scene description text respectively; adding the scene layout information and the style category to the initial scene description text respectively to obtain a target scene description text; obtaining an initial noise image and obtaining first image features of the noise image; performing convolution processing on text semantic features of the target scene description text and the first image features to obtain second image features; decoding the second image features to obtain a target image after noise reduction for the noise image; The target image is used to present the scene described by the scene description text; position relationships between scene objects in the presented scene meet the relative position relationships.

2. The method of claim 1, wherein, The scene layout information comprises a scene layout category; the prediction of the scene layout information corresponding to the scene description text according to the text semantic features of the scene description text comprises: performing layout classification processing on the scene description text according to the text semantic features of the scene description text to obtain a scene layout category to which the scene description text belongs.

3. The method of claim 1, wherein, The classification of the creative text according to the text semantic features of the creative text to obtain the style category to which the creative text belongs comprises: determining the creative text to which the initial scene description text belongs; the creative text comprises a plurality of scene description texts for describing scenes created by the creative text; extracting text semantic features of the creative text according to the scene description texts in the creative text; performing style classification processing on the creative text according to the text semantic features of the creative text to obtain the style category to which the creative text belongs.

4. The method of claim 1, wherein, The convolution processing of the text semantic features of the target scene description text and the first image features to obtain the second image features comprises: diffusion sampling is performed on the first image features of the noise image at a first encoding time to obtain the first image features of the noise image at a second encoding time; the first encoding time is located before the second encoding time; taking the first image features of the noise image at the second encoding time as reference image features to be noise-reduced in the current round, taking the second encoding time as a current encoding time, and performing convolution processing on the text semantic features of the target scene description text and the reference image features in the current round to obtain an intermediate state image feature after noise reduction corresponding to the current encoding time in the current round; The previous encoding time of the current encoding time is taken as a current encoding time of a next round, the next round is taken as the current round, and the intermediate state image feature is taken as a reference image feature to be denoised in the current round, so as to perform iterative convolution processing until the current encoding time is the first encoding time, and the iterative denoising is stopped, and the intermediate state image feature corresponding to the first encoding time is taken as the second image feature.

5. The method of claim 1, wherein, The scene layout information is determined by the trained layout recognition model; the method further comprises: Obtaining at least one first sample scene description text; the first sample scene description text is labeled with reference scene layout information; Performing layout prediction on the first sample scene description text by the to-be-trained layout recognition model to obtain predicted scene layout information; According to the difference between the predicted scene layout information and the reference scene layout information, the to-be-trained layout recognition model is iteratively trained to obtain the trained layout recognition model.

6. The method of claim 1, wherein, The target image is generated by the trained image generation model; the method further comprises: Obtaining at least one sample image-text pair; the sample image-text pair includes a second sample scene description text and a reference image; the second sample scene description text adds scene layout information corresponding to the second sample scene description text; the reference image presents the scene described in the second sample scene description text; For each sample image-text pair, noise is added to the reference image in the sample image-text pair to obtain a sample noise image, and the sample noise image and the second sample scene description text in the sample image-text pair are input into a to-be-trained image generation model, so that the to-be-trained image generation model denoises the sample noise image based on the second sample scene description text to obtain a predicted image corresponding to the sample image-text pair; According to the difference between the predicted image corresponding to each sample image-text pair and the corresponding reference image, the to-be-trained image generation model is iteratively trained to obtain the trained image generation model.

7. The method of claim 6, wherein, The second sample scene description text further adds a style category corresponding to the second sample scene description text; the sample image-text pair used in each iteration in the iterative training process includes at least one batch of sample image-text pairs; the reference image in each batch of sample image-text pairs includes a reference layout image and a reference style image; According to the difference between the predicted image corresponding to each sample image-text pair and the corresponding reference image, the to-be-trained image generation model is iteratively trained to obtain the trained image generation model, comprising: For each batch in each iteration training, the difference between the predicted image corresponding to each sample image-text pair in the batch and the corresponding reference image is used to train the to-be-trained image generation model to obtain an image generation model corresponding to the batch; In the case of meeting the iteration stop condition, the image generation model corresponding to the last batch in the last iteration training is taken as the trained image generation model.

8. The method of claim 7, wherein, The reference image in each batch further includes a sample style transfer image; the sample style transfer image is an image obtained by performing migration processing on the reference layout image for different styles; the sample style transfer image has a layout corresponding to the reference layout image and a style used in the migration processing.

9. The method of claim 7, wherein, The reference layout image in each sample image-text pair in each batch is an image extracted from a pre-generated layout image group; images in the same layout image group have the same layout; The method further includes: obtaining at least one video; performing foreground erasing on each image in each video respectively to obtain a background image; performing clustering on the background images according to image layout features corresponding to the background images respectively to obtain at least one layout image group.

10. The method of claim 9, wherein, The method further includes: performing saliency identification on each image in each video to obtain a saliency image of the image; a gray value of each pixel point in the saliency image is used to represent a saliency degree of the pixel point; generating a mask image according to the gray values of the pixel points in the saliency image respectively; the mask image is used to represent a saliency region in the image; erasing the saliency region in the image according to the mask image to obtain a background image corresponding to the image.

11. The method of claim 7, wherein, The reference style image in each sample image-text pair in each batch is an image extracted from a pre-generated style image group; images in the same style image group have the same style; the method further includes: obtaining at least one video; wherein one video corresponds to one style; sampling at least one video image from each video and combining the sampled video images to obtain a style image group corresponding to the video.

12. The method of claim 6, wherein, The method further includes: for each sample image-text pair, determining a first loss value according to a difference between a prediction image feature of the prediction image corresponding to the sample image-text pair and a reference image feature of the corresponding reference image; determining a second loss value according to a difference between pixel values of pixel points in the prediction image corresponding to the sample image-text pair and pixel values of pixel points in the corresponding reference image; fusing the first loss value and the second loss value to obtain a target loss value corresponding to the sample image-text pair; performing iterative training on the image generation model to be trained according to the target loss values corresponding to the sample image-text pairs respectively to obtain a trained image generation model.

13. An image generation apparatus characterized by comprising: The device includes: an obtaining module configured to disassemble the creative text to obtain a plurality of initial scene description texts; The prediction module is configured to predict scene layout information corresponding to the scene description text according to a text semantic feature of the scene description text; the scene layout information is used to represent relative position relationships between scene objects in a scene described by the scene description text. The determination module is configured to perform style classification on the creative text according to a text semantic feature of the creative text, to obtain a style category to which the creative text belongs, and to take the style category to which the creative text belongs as the style category of the initial scene description text. The addition module is configured to add the scene layout information and the style category into the initial scene description text respectively, to obtain a target scene description text. The obtaining module is further configured to obtain an initial noise image and to obtain a first image feature of the noise image. The denoising module is configured to perform convolution processing on the text semantic feature of the target scene description text and the first image feature, to obtain a second image feature; and to perform decoding on the second image feature, to obtain a target image obtained by denoising the noise image. The target image is used to present the scene described by the scene description text; position relationships between scene objects in the presented scene satisfy the relative position relationships.

14. The image generation apparatus according to claim 13, wherein The scene layout information includes a scene layout category; and the prediction module is further configured to perform layout classification processing on the scene description text according to a text semantic feature of the scene description text, to obtain a scene layout category to which the scene description text belongs.

15. The image generation apparatus of claim 13, wherein The determination module is further configured to determine a creative text to which the initial scene description text belongs; the creative text includes a plurality of created scene description texts for describing a scene; to extract a text semantic feature of the creative text according to the scene description texts in the creative text; and to perform style classification processing on the creative text according to the text semantic feature of the creative text, to obtain a style category to which the creative text belongs.

16. The image generation apparatus according to claim 13, wherein The denoising module is further configured to perform diffusion sampling on a first image feature of the noise image at a first encoding time, to obtain a first image feature of the noise image at a second encoding time; The first encoding time is located before the second encoding time; the first image feature of the noise image at the second encoding time is taken as a reference image feature to be denoised in a current round, the second encoding time is taken as a current encoding time, convolution processing is performed on the text semantic feature of the target scene description text and the reference image feature in the current round, to obtain an intermediate state image feature of the noise image after denoising in the current encoding time corresponding to the current round; a previous encoding time of the current encoding time is taken as a current encoding time of a next round, the next round is taken as the current round, and the intermediate state image feature is taken as the reference image feature to be denoised in the current round, to perform iterative convolution processing, until the current encoding time is the first encoding time, and the iterative denoising is stopped; and an intermediate state image feature corresponding to the first encoding time is taken as the second image feature.

17. The image generation apparatus of claim 13, wherein The device further comprises a first training module configured to obtain at least one first sample scene description text, wherein the first sample scene description text is labeled with reference scene layout information; perform layout prediction on the first sample scene description text by using a to-be-trained layout recognition model to obtain predicted scene layout information; and perform iterative training on the to-be-trained layout recognition model according to the difference between the predicted scene layout information and the reference scene layout information, to obtain a trained layout recognition model.

18. The image generation apparatus of claim 13, wherein The device further comprises a second training module configured to obtain at least one sample image-text pair, wherein the sample image-text pair comprises a second sample scene description text and a reference image, the second sample scene description text is added with scene layout information corresponding to the second sample scene description text, and the reference image is used to present a scene described in the second sample scene description text; for each sample image-text pair, add noise to the reference image in the sample image-text pair to obtain a sample noise image, and input the sample noise image and the second sample scene description text in the sample image-text pair into a to-be-trained image generation model, so as to perform noise reduction on the sample noise image based on the second sample scene description text by using the to-be-trained image generation model, to obtain a predicted image corresponding to the sample image-text pair. Perform iterative training on the to-be-trained image generation model according to the difference between the predicted image corresponding to each sample image-text pair and the corresponding reference image, to obtain a trained image generation model.

19. The image generation means of claim 18, wherein, The second sample scene description text is further added with a style category corresponding to the second sample scene description text; the sample image-text pair used in each iteration in the iterative training process comprises at least one batch of sample image-text pairs; the reference image in each batch of sample image-text pairs comprises a reference layout image and a reference style image. The second training module is further configured to, for each batch in each iteration, perform training on the to-be-trained image generation model according to the difference between the predicted image corresponding to each sample image-text pair in the batch and the corresponding reference image, to obtain an image generation model corresponding to the batch; and in a case where an iteration stop condition is met, take the image generation model corresponding to the last batch in the last iteration as the trained image generation model.

20. The image generation means of claim 19, wherein, The reference image in each batch further comprises a sample style transfer image; the sample style transfer image is an image obtained by performing style transfer processing on the reference layout image; and the sample style transfer image has the layout corresponding to the reference layout image and the style used in the style transfer processing.

21. The image generation means of claim 19, wherein, Each batch of the sample image-text pair refers to a layout image extracted from a pre-generated layout image group; images in the same layout image group have the same layout; the device further comprises a first generation module, which is configured to obtain at least one video; perform foreground erasing on each image in each video to obtain a background image; cluster the background images according to the image layout features corresponding to the background images, to obtain at least one layout image group.

22. The image generation device of claim 21, wherein, The first generation module is further configured to, for each image in each video, perform saliency identification on the image to obtain a saliency image of the image; a gray value of each pixel point in the saliency image is used to represent the saliency degree of the pixel point; generate a mask image according to the gray values of the pixel points in the saliency image; the mask image is used to represent a saliency region in the image; perform erasing on the saliency region in the image according to the mask image to obtain a background image corresponding to the image.

23. The image generation means of claim 19, wherein, Each batch of the sample image-text pair refers to a style image extracted from a pre-generated style image group; images in the same style image group have the same style; the device further comprises a second generation module, which is configured to obtain at least one video; wherein one video corresponds to one style; for each video, sample at least one video image from the video, and combine the sampled video images to obtain a style image group corresponding to the video.

24. The image generation apparatus of claim 18, wherein The second training module is further configured to, for each sample image-text pair, determine a first loss value according to the difference between the predicted image features of the predicted image corresponding to the sample image-text pair and the reference image features of the corresponding reference image; determine a second loss value according to the difference between the pixel values of the pixel points in the predicted image corresponding to the sample image-text pair and the pixel values of the pixel points in the corresponding reference image; and fuse the first loss value and the second loss value to obtain a target loss value corresponding to the sample image-text pair. Iteratively train the image generation model to be trained according to the target loss values corresponding to the sample image-text pairs, to obtain a trained image generation model. 25.A computer device, comprising a memory and a processor, wherein the memory stores a computer program. The processor executes the computer program to implement the method of any one of claims 1 to 12.

26. A computer readable storage medium storing a computer program, wherein the computer program comprises instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-25. The computer program is executed by the processor to implement the method of any one of claims 1 to 12.

27. A computer program product comprising a computer program, characterised in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 12. The computer program is executed by the processor to implement the method of any one of claims 1 to 12.

Images