Image generation method and device, equipment and storage medium

Abstract

The invention provides an image generation method and device, equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: inputting a current text and a previous image corresponding to a previous text into an autoregression module in an image generation model, and obtaining a current token sequence outputted by the autoregression module; inputting the previous image into an image feature adaptation module in the image generation model to obtain image features output by the image feature adaptation module; and based on the current token sequence and the image features, determining a current image corresponding to the current text. According to the invention, the consistency of images generated before and after can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image generation method, device, equipment and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, the application of literary images is becoming more and more extensive. In fields such as picture book generation, it is usually necessary to use multiple consecutive images to depict a complete story. This requires the generated images to have continuity, including the consistency of the image of the story characters, the consistency of the scenes, and the consistency of the image style.

[0003] In the prior art, the consistency of the generated image is usually ensured by editing the text content based on the diffusion model. For example, the description of the character image is kept consistent so that the character in the generated image is as similar as possible in appearance, and the stylized description is made in the text so that the generated image is as close in style as possible. In addition, the random number sampling is kept consistent so that the previous and next images are as close in content as possible.

[0004] However, since the description of the story content in picture books is always changing and developing, and the diffusion model is highly sensitive to text, when the content of the text changes, the generated image will be inconsistent, resulting in poor image consistency. Summary of the invention

[0005] The present invention provides an image generation method, device, equipment and storage medium, which are used to solve the defect in the prior art that when the text content changes, the consistency of the images generated before and after is not high, and to improve the consistency of the images generated before and after.

[0006] The present invention provides an image generation method, comprising: Input the current text and the previous image corresponding to the previous text into the autoregressive module in the image generation model to obtain the current token sequence output by the autoregressive module; Inputting the previous image into the image feature adaptation module in the image generation model to obtain image features output by the image feature adaptation module; Based on the current token sequence and the image features, a current image corresponding to the current text is determined.

[0007] According to an image generation method provided by the present invention, the determining of the current image corresponding to the current text based on the current token sequence and the image features includes: Inputting the current token sequence and the image features into a decoupled cross attention module in the image generation model to obtain text attention features and image attention features output by the decoupled cross attention module; The text attention feature and the image attention feature are input into a decoder in the image generation model to obtain the current image output by the decoder.

[0008] According to an image generation method provided by the present invention, the decoder includes a diffusion model.

[0009] According to an image generation method provided by the present invention, the decoder is trained based on the following method: Inputting the sample image into the image feature extraction module to obtain the sample image features output by the image feature extraction module; Inputting the sample image features into an initial decoder to obtain a first predicted image; Based on the first predicted image and the sample image, the model parameters of the initial decoder are iteratively optimized to obtain the decoder.

[0010] According to an image generation method provided by the present invention, the decoupled cross attention module is trained based on the following method: Inputting the sample image feature of the sample image and the sample text token sequence corresponding to the sample text into an initial decoupled cross attention module to obtain a first sample text attention feature and a first sample image attention feature output by the initial decoupled cross attention module; Inputting the first sample text attention feature and the first sample image attention feature into the decoder to obtain a second predicted image corresponding to the sample text output by the decoder; Based on the second predicted image and the sample image, the model parameters of the initial decoupled cross-attention module are iteratively optimized to obtain the decoupled cross-attention module.

[0011] According to an image generation method provided by the present invention, the autoregressive module is trained based on the following method: Inputting the sample text and the sample image into an initial autoregressive module to obtain a sample token sequence output by the initial autoregressive module; Inputting the sample token sequence and the sample image feature into the decoupled cross attention module to obtain a second sample text attention feature and a second sample image attention feature output by the decoupled cross attention module; Inputting the second sample text attention feature and the second sample image attention feature into the decoder to obtain a third predicted image corresponding to the sample text output by the decoder; Based on the third predicted image and the sample image, the model parameters of the initial autoregressive module are iteratively optimized to obtain the autoregressive module.

[0012] The present invention also provides an image generating device, comprising: An input module, used to input the current text and the previous image corresponding to the previous text into the autoregressive module in the image generation model to obtain the current token sequence output by the autoregressive module; The input module is further used to input the previous image into the image feature adaptation module in the image generation model to obtain the image features output by the image feature adaptation module; A determination module is used to determine a current image corresponding to the current text based on the current token sequence and the image features.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned image generation methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image generating method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image generating method described above is implemented.

[0016] The image generation method, device, equipment and storage medium provided by the present invention input the current text and the previous image corresponding to the previous text into the autoregressive module in the image generation model to obtain the current token sequence output by the autoregressive module, and input the previous image into the image feature adaptation module in the image generation model to obtain the image features output by the image feature adaptation module, and then determine the current image corresponding to the current text based on the current token sequence and the image features. Since when generating the current image, it is not based solely on the current text, but guided by the previous image, and the image features of the previous image are used as a guide to generate the current image, the generated current image can be made similar to the previous image, so that the consistency between the generated previous and next images can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of a flow chart of an image generation method provided in an embodiment of the present invention.

[0019] Figure 2 A schematic diagram of the structure of an autoregressive network provided by an embodiment of the present invention.

[0020] Figure 3 This is one of the structural schematic diagrams of the image generation model provided by an embodiment of the present invention.

[0021] Figure 4 The second structural diagram of the image generation model provided by the embodiment of the present invention.

[0022] Figure 5 A schematic diagram of the structure of an image generating device provided by an embodiment of the present invention.

[0023] Figure 6 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] With the development of artificial intelligence (AI) technology, the application of Wenshengtu is becoming more and more extensive. Among them, picture book generation is a practical application scenario of Wenshengtu. It is mainly aimed at children's education and has a wide range of needs in the fields of learning machines or children's book publishing. Picture books are a way of presenting a children's story in a concrete way through cartoon drawings. Picture books often need to depict a complete story through multiple consecutive images. Therefore, it is necessary to ensure the consistency of the image of the story characters, the consistency of the scenes, and the consistency of the image style. Therefore, it is very important to generate images with consistent features.

[0026] In the prior art, text is usually input into a diffusion model to generate an image, and the consistency of the generated image is ensured by editing the content of the text. For example, the description of the character image is kept consistent so that the characters in the generated image are as similar as possible in appearance, and a stylized description is made in the text so that the style of the generated image is as close as possible. In addition, the random number sampling is kept consistent so that the content of the previous and next images is as close as possible. However, despite the consistency of the text description, since the description of the story content of the picture book is always changing and developing, and the diffusion model is highly sensitive to the text, the consistency of some texts can easily be diluted as a whole, making the consistency of the generated image uncontrollable, resulting in inconsistency of the generated images.

[0027] In view of the above problems, an embodiment of the present invention provides an image generation method, in which, in addition to the current text, the previous image generated according to the previous text can also be input into the autoregressive module in the image generation model to obtain the current token sequence, and the previous image is input into the image feature adaptation module in the image generation model to obtain the image features output by the image feature adaptation module, so as to use the image features corresponding to the previous image as a guide and generate the current image corresponding to the current text based on the current token sequence. Since the previous image is used as a guide when generating the current image, the image features of the previous image are used as a standard to generate the current image, so the consistency between the generated previous and next images can be improved.

[0028] Combine the following Figures 1 to 4 The image generation method provided by the embodiment of the present invention is described. The execution subject of the method can be an electronic device such as a terminal device, a computer or a server, or a smart device specially designed for image generation, or an image generation device set in the electronic device or smart device, and the image generation device can be implemented by software, hardware or a combination of the two. The method can be applied to any scene where a text image needs to be produced, especially to scenes where the image style needs to be consistent, such as picture book generation, game design or film and television production.

[0029] Figure 1 A schematic diagram of a flow chart of an image generation method provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes: Step 101: Input the current text and the previous image corresponding to the previous text into the autoregressive module in the image generation model to obtain the current token sequence output by the autoregressive module.

[0030] In this step, in the text-image scenario, a paragraph of text usually corresponds to at least one image. The current text can be understood as a paragraph of text that currently needs to generate a corresponding image, and the previous text is the text based on which the image was generated last time. It should be understood that the moment when the image needs to be generated currently can be determined as the current moment, and the moment when the image was generated last time can be determined as the previous moment. Therefore, the previous image corresponding to the previous text can also be understood as the image generated at the previous moment.

[0031] The autoregressive module can use Emu2 or other autoregressive networks. Figure 2 A schematic diagram of the structure of an autoregressive network provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown in the figure, the autoregressive network includes an encoder, a generative multimodal model and a decoder. The encoder is an image encoder network based on contrastive language-image pre-training (CLIP). After the current text and the previous image corresponding to the previous text are input into the encoder of the autoregressive module, the encoder will encode the features of the current text and the previous image, and input the encoded features into the generative multimodal model. The current token sequence containing semantic information output by the autoregressive module is obtained by autoregression. The current token sequence can be used as the input of the decoder. The decoder can be an SDXL network or other network that can decode tokens. The decoder decodes tokens based on classification information.

[0032] Step 102: Input the previous image into the image feature adaptation module in the image generation model to obtain the image features output by the image feature adaptation module.

[0033] In this step, the image feature adaptation module can be used to extract image features from the previous image to guide the subsequent generation of the current image, so that the newly generated current image can maintain feature consistency with the previous image.

[0034] Specifically, after the previous image is input into the image feature adaptation module, the feature information in the previous image is extracted through the image encoder in the image feature adaptation module. These feature information can be used to represent the image content or image style, etc., and then transformed through the linear transformation layer in the image feature adaptation module to combine the feature vectors output by the image encoder to achieve the purpose of adjusting the feature dimension. In addition, the features after linear transformation can also be input into the normalization layer for normalization processing to obtain the image features of the previous image.

[0035] Step 103: Based on the current token sequence and image features, determine the current image corresponding to the current text.

[0036] In this step, after determining the current token sequence and image features, the current image corresponding to the current text can be generated by decoding the current token sequence based on the current token sequence and image features, with the image features as a guide. The image features are used as a single prompt, and a layer dedicated to processing image features is set in each cross-attention layer through a decoupled cross-attention method. In this way, the image features and the text features corresponding to the current token sequence can be processed separately through their respective cross-attention layers, avoiding the information loss that may be caused by direct merging.

[0037] The image generation method provided by the embodiment of the present invention inputs the current text and the previous image corresponding to the previous text into the autoregressive module in the image generation model to obtain the current token sequence output by the autoregressive module, and inputs the previous image into the image feature adaptation module in the image generation model to obtain the image features output by the image feature adaptation module, and then determines the current image corresponding to the current text based on the current token sequence and the image features. Since when generating the current image, it is not based solely on the current text, but guided by the previous image, and the image features of the previous image are used as a guide to generate the current image, the generated current image can be made more similar to the previous image, and therefore, the consistency between the generated previous and next images can be improved.

[0038] In the picture book application scenario, the autoregressive model can make the picture book generation process have time series characteristics. In addition to text information, the previous image generated by the previous scene can also be used as guidance information for the next scene, so that the previous and next scenes and stories depicted in the picture book have continuity, and make up for the lack of text information in the current scene, enriching the details of the current scene. In addition, the image feature adaptation module (Image Prompt Adapter, IP-Adapter) is used to enhance the consistency of the image of the story characters. IP-Adapter extracts the visual features of the characters as guidance information, and guides the generation of the current image together with the text information, thereby generating a current image containing the guidance information, further ensuring the consistency of the current image and the previous image in terms of characters and content.

[0039] Exemplarily, based on the above embodiment, when determining the current image corresponding to the current text based on the current token sequence and image features, it can be done in the following manner: The current token sequence and image features are input into the decoupled cross-attention module in the image generation model to obtain the text attention features and image attention features output by the decoupled cross-attention module, and the text attention features and image attention features are input into the decoder in the image generation model to obtain the current image output by the decoder.

[0040] Specifically, Figure 3 One of the structural diagrams of the image generation model provided by the embodiment of the present invention is as follows: Figure 3 As shown, the image generation model includes an autoregressive module, an image feature adaptation module and a decoder, wherein the current text "The kitten found a cave" and the previous image corresponding to the previous text are input into the autoregressive module to obtain the current token sequence, wherein the text corresponding to the previous image is "A kitten wearing a hat walks in the forest". The previous image is input into the image feature adaptation module IP-Adapter to obtain the image features, and the current token sequence and image features are input into the decoder to obtain the current image corresponding to the current text. Among them, Figure 3 The t=n in it represents the current time, and t=n-1 represents the time when the image was last generated.

[0041] Figure 4 The second structural diagram of the image generation model provided by the embodiment of the present invention is as follows: Figure 4 As shown in the figure, the current text "a little girl wearing glasses" and the previous image corresponding to the previous text are input into the autoregressive network, so that the current token sequence output by the autoregressive network can be obtained.

[0042] In addition, the previous image corresponding to the previous text is input into the image feature adaptation module, and the image features of the previous image are obtained through the image decoder, linear transformation and normalization processing. The obtained current token sequence and the image features of the previous image can also be input into the decoupled cross-attention module of the image generation model. Among them, the decoupled cross-attention module includes two cross-attention modules, which process image features and text features respectively, so as to obtain text attention features and image attention features output by each cross-attention module. Since the image attention features and text attention features are processed by their respective cross-attention modules, the image generation model can independently focus on important information in the image and text.

[0043] Furthermore, the obtained text attention features and image attention features are input into the decoder in the image generation model to obtain the current image output by the decoder. The decoder can use a diffusion model U-Net model, such as SDXL, to convert the noise x at time t into t, text attention features and image attention features are input into the decoder U-Net model together. Through step-by-step denoising, in each denoising process, the denoising U-Net model uses image features and text features to guide the generation of the image, thereby obtaining the final clear image. It should be understood that the denoising U-Net model obtains the latent of the image after decoding, and can also be decoded by a pre-trained variational autoencoder (VAE) to obtain the final current image.

[0044] Since the decoder is set as a diffusion model, the final current image is generated by a step-by-step denoising method. In each denoising process, the image generation process can be more finely controlled, thereby improving the image quality of the current image.

[0045] Among them, the above-mentioned decoupled cross-attention module can be set in the image feature adaptation module, can be set in the decoder, or can be set separately.

[0046] It should be noted that at the initial moment, that is, when the image is generated for the first time, since there is no previous image corresponding to the previous text, only the text content is input into the autoregressive module, and the image tokens are generated by the autoregressive module, and the image tokens are input into the decoder for decoding to generate the corresponding image. At this time, the input of the image feature adapter module IP-Adapter is empty. At any subsequent moment, the image tokens corresponding to the current text and the previous image can be input into the autoregressive module to obtain the tokens at the current moment, and the previous image can be used as the standard input image feature adapter module IP-Adapter for feature extraction, thereby guiding the decoder to generate a current image with consistent features.

[0047] In addition, the character image can also be customized, and any character image can be input into the image feature adaptation module IP-Adapter as a guide image, so that the character image in the current image finally generated can be controlled.

[0048] In this embodiment, the image feature of the previous image is separately proposed as a prompt feature. By setting a decoupled cross-attention module, and setting a separate text cross-attention module and an image cross-attention module in the decoupled cross-attention module, the text attention feature and the image attention feature are distinguished. Therefore, the image generation model can independently process the image feature and the text feature, reducing the loss of information, thereby improving the correlation between the image and the text description. In the picture book application scenario, the consistency of the picture book content can be maintained in the above manner. In addition, by generating the current token sequence through the autoregressive model and generating the current image through the current token sequence, the consistency of the generated image scene and the image style can be guaranteed. The consistency of the character image can be guaranteed through the image feature adaptation module, and the character image can be customized. Moreover, when the image generation method of the present invention is used to generate the image, the production efficiency can be improved for the offline picture book generation scenario, and the generation effect and quality can be improved for the online picture book generation scenario.

[0049] Exemplarily, based on the above embodiments, the training process of each module in the image generation model is described in detail below.

[0050] The decoder is trained based on the following method: The sample image is input into the image feature extraction module to obtain the sample image features output by the image feature extraction module, the sample image features are input into the initial decoder to obtain the first predicted image, and based on the first predicted image and the sample image, the model parameters of the initial decoder are iteratively optimized to obtain the decoder.

[0051] Specifically, multiple sample images are collected by a camera device or obtained from an image database, and each sample image is input into an image feature extraction module, such as a convolutional neural network (CNN), to obtain sample image features output by the image feature extraction module. The extracted sample image features are then input into the initial decoder U-Net, and the sample image features are decoded by the initial decoder U-Net to obtain the first predicted image output by the initial decoder. Based on the difference between the obtained first predicted image and the sample image, the loss information is determined, and the model parameters of the initial decoder are adjusted according to the loss information, and the above process is iteratively performed until the loss is minimized or the obtained model converges, and the final model is determined as the decoder.

[0052] In this embodiment, since the model parameters of the initial decoder can be iteratively optimized based on the first predicted image and the sample image to train the decoder, when the current image is generated based on the decoder finally obtained, the generated current image can be made consistent in style with the previous input image, thereby improving the consistency between the current image and the previously generated image.

[0053] The decoupled cross attention module is trained based on the following method: The sample image features of the sample image and the sample text token sequence corresponding to the sample text are input into the initial decoupled cross-attention module to obtain the first sample text attention features and the first sample image attention features output by the initial decoupled cross-attention module, and the first sample text attention features and the first sample image attention features are input into the decoder to obtain the second predicted image corresponding to the sample text output by the decoder, and based on the second predicted image and the sample image, the model parameters of the initial decoupled cross-attention module are iteratively optimized to obtain the decoupled cross-attention module.

[0054] Specifically, the sample text is text related to the text used to generate the sample image, such as two adjacent paragraphs of text in a story. The sample text is input into the text encoder to obtain the sample text token sequence output by the text encoder. The sample image features and the sample text token sequence extracted from each sample image are input into the initial decoupled cross-attention module, which processes the image features and text features separately, such as processing the image and text respectively through different cross-attention modules, thereby obtaining the first sample text attention feature and the first sample image attention feature output by the initial decoupled cross-attention module. By processing the image and text separately, the interference between different modal features can be reduced, so that the model can focus more on the key information of each modality.

[0055] Furthermore, the obtained first sample text attention feature and the first sample image attention feature can be input into a previously trained decoder, and the first sample text attention feature and the first sample image attention feature can be decoded by the decoder to obtain a second predicted image corresponding to the sample text. Loss information is determined based on the difference between the second predicted image and the sample image, and the model parameters of the initial decoupled cross attention module are iteratively optimized based on the loss information, thereby obtaining a finally trained decoupled cross attention module to minimize the difference between the second predicted image and the sample image, so that when the decoupled cross attention module is used to generate the current image, the difference between the generated current image and the previous image can be reduced.

[0056] In this embodiment, the image features and text features are processed separately by the initial decoupled cross-attention module, and the initial decoupled cross-attention module is trained based on the obtained first sample text attention features and the first sample image attention features, so that the decoupled cross-attention module obtained after training can achieve stronger generalization ability when facing new text or images. In addition, by decoupling the features of the sample text and the features of the sample image, information loss can be reduced and more key information can be retained, thereby improving the accuracy of the trained decoupled cross-attention module.

[0057] Among them, the autoregressive module is trained based on the following method: The sample text and the sample image are input into the initial autoregressive module to obtain the sample token sequence output by the initial autoregressive module, and the sample token sequence and the sample image features are input into the decoupled cross-attention module to obtain the second sample text attention feature and the second sample image attention feature output by the decoupled cross-attention module, and the second sample text attention feature and the second sample image attention feature are input into the decoder to obtain the third predicted image corresponding to the sample text output by the decoder, and then the model parameters of the initial autoregressive module are iteratively optimized based on the third predicted image and the sample image to obtain the autoregressive module.

[0058] Specifically, after training the decoder and the decoupled cross-attention module, the autoregressive module can be further trained. The initial autoregressive module is usually a Transformer-based model. After inputting the sample text and sample image into the initial autoregressive module, the corresponding sample token sequence can be generated. The sample token sequence is the encoded representation of the sample text.

[0059] The obtained sample token sequence and sample image features are input into the trained decoupled cross attention module, which processes the image features and text features separately to generate the second sample text attention features and the second sample image attention features. Processing image features and text features separately can reduce the interference between features of different modalities, allowing the model to focus more on the key information of each modality.

[0060] The second sample text attention feature and the second sample image attention feature are input into the trained decoder, and the decoder generates a third predicted image corresponding to the sample text according to these features. Based on the difference between the third predicted image and the sample image, the loss information is determined, and the model parameters of the initial autoregressive module are iteratively optimized according to the loss information to obtain the autoregressive module. Since the difference between the third predicted image and the sample image is used as the loss information, the difference between the third predicted image and the sample image can be minimized.

[0061] In this embodiment, the sample text and sample image are input into the initial autoregressive module, and then the generated sample token sequence and sample image features are input into the decoupled cross-attention module, and then the second sample text attention features and the second sample image attention features output by the decoupled cross-attention module are input into the decoder, and finally the autoregressive module is iteratively optimized based on the difference between the obtained third predicted image and the sample image. The final autoregressive module can more accurately generate an image that matches the text description, thereby improving the quality of image generation.

[0062] The image generating device provided by the present invention is described below. The image generating device described below and the image generating method described above can be referred to each other.

[0063] Figure 5 A schematic diagram of the structure of an image generating device provided by an embodiment of the present invention, referring to Figure 5 As shown, the image generating device 500 includes: An input module 11 is used to input the current text and the previous image corresponding to the previous text into the autoregressive module in the image generation model to obtain the current token sequence output by the autoregressive module; The input module 11 is further used to input the previous image into the image feature adaptation module in the image generation model to obtain the image features output by the image feature adaptation module; The determination module 12 is used to determine the current image corresponding to the current text based on the current token sequence and the image features.

[0064] In an exemplary embodiment, the determination module 12 is specifically configured to: Inputting the current token sequence and the image features into a decoupled cross attention module in the image generation model to obtain text attention features and image attention features output by the decoupled cross attention module; The text attention feature and the image attention feature are input into a decoder in the image generation model to obtain the current image output by the decoder.

[0065] In an example embodiment, the decoder includes a diffusion model.

[0066] In an example embodiment, the decoder is trained based on the following method: Inputting the sample image into the image feature extraction module to obtain the sample image features output by the image feature extraction module; Inputting the sample image features into an initial decoder to obtain a first predicted image; Based on the first predicted image and the sample image, the model parameters of the initial decoder are iteratively optimized to obtain the decoder.

[0067] In an exemplary embodiment, the decoupled cross attention module is trained based on the following method: Inputting the sample image feature of the sample image and the sample text token sequence corresponding to the sample text into an initial decoupled cross attention module to obtain a first sample text attention feature and a first sample image attention feature output by the initial decoupled cross attention module; Inputting the first sample text attention feature and the first sample image attention feature into the decoder to obtain a second predicted image corresponding to the sample text output by the decoder; Based on the second predicted image and the sample image, the model parameters of the initial decoupled cross-attention module are iteratively optimized to obtain the decoupled cross-attention module.

[0068] In an exemplary embodiment, the autoregressive module is trained based on the following method: Inputting the sample text and the sample image into an initial autoregressive module to obtain a sample token sequence output by the initial autoregressive module; Inputting the sample token sequence and the sample image feature into the decoupled cross attention module to obtain a second sample text attention feature and a second sample image attention feature output by the decoupled cross attention module; Inputting the second sample text attention feature and the second sample image attention feature into the decoder to obtain a third predicted image corresponding to the sample text output by the decoder; Based on the third predicted image and the sample image, the model parameters of the initial autoregressive module are iteratively optimized to obtain the autoregressive module.

[0069] The device of this embodiment can be used to execute the method of any embodiment in the image generation method side embodiment. Its specific implementation process and technical effects are similar to those in the image generation method side embodiment. For details, please refer to the detailed introduction in the image generation method side embodiment, which will not be repeated here.

[0070] Figure 6 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the image generation method, which includes: inputting the current text and the previous image corresponding to the previous text into the autoregressive module in the image generation model to obtain the current token sequence output by the autoregressive module; inputting the previous image into the image feature adaptation module in the image generation model to obtain the image features output by the image feature adaptation module; and determining the current image corresponding to the current text based on the current token sequence and the image features.

[0071] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0072] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image generation method provided by the above-mentioned methods, which includes: inputting the current text and the previous image corresponding to the previous text into the autoregressive module in the image generation model to obtain the current token sequence output by the autoregressive module; inputting the previous image into the image feature adaptation module in the image generation model to obtain the image features output by the image feature adaptation module; and determining the current image corresponding to the current text based on the current token sequence and the image features.

[0073] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image generation method provided by the above-mentioned methods, the method comprising: inputting a current text and a previous image corresponding to a previous text into an autoregressive module in an image generation model to obtain a current token sequence output by the autoregressive module; inputting the previous image into an image feature adaptation module in the image generation model to obtain image features output by the image feature adaptation module; and determining a current image corresponding to the current text based on the current token sequence and the image features.

[0074] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0075] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image generation method, characterized in that: include: Input the current text and the previous image corresponding to the previous text into the autoregressive module in the image generation model to obtain the current token sequence output by the autoregressive module; Inputting the previous image into the image feature adaptation module in the image generation model to obtain image features output by the image feature adaptation module; Based on the current token sequence and the image features, a current image corresponding to the current text is determined.

2. The image generation method according to claim 1, characterized in that: The determining, based on the current token sequence and the image feature, a current image corresponding to the current text includes: Inputting the current token sequence and the image features into a decoupled cross attention module in the image generation model to obtain text attention features and image attention features output by the decoupled cross attention module; The text attention feature and the image attention feature are input into a decoder in the image generation model to obtain the current image output by the decoder.

3. The image generation method according to claim 2, characterized in that: The decoder includes a diffusion model.

4. The image generation method according to claim 2 or 3, characterized in that: The decoder is trained based on the following method: Inputting the sample image into the image feature extraction module to obtain the sample image features output by the image feature extraction module; Inputting the sample image features into an initial decoder to obtain a first predicted image; Based on the first predicted image and the sample image, the model parameters of the initial decoder are iteratively optimized to obtain the decoder.

5. The image generation method according to claim 4, characterized in that: The decoupled cross attention module is trained based on the following method: Inputting the sample image feature of the sample image and the sample text token sequence corresponding to the sample text into an initial decoupled cross attention module to obtain a first sample text attention feature and a first sample image attention feature output by the initial decoupled cross attention module; Inputting the first sample text attention feature and the first sample image attention feature into the decoder to obtain a second predicted image corresponding to the sample text output by the decoder; Based on the second predicted image and the sample image, the model parameters of the initial decoupled cross-attention module are iteratively optimized to obtain the decoupled cross-attention module.

6. The image generation method according to claim 5, characterized in that: The autoregressive module is trained based on the following method: Inputting the sample text and the sample image into an initial autoregressive module to obtain a sample token sequence output by the initial autoregressive module; Inputting the sample token sequence and the sample image feature into the decoupled cross attention module to obtain a second sample text attention feature and a second sample image attention feature output by the decoupled cross attention module; Inputting the second sample text attention feature and the second sample image attention feature into the decoder to obtain a third predicted image corresponding to the sample text output by the decoder; Based on the third predicted image and the sample image, the model parameters of the initial autoregressive module are iteratively optimized to obtain the autoregressive module.

7. An image generating device, characterized in that: include: An input module, used to input the current text and the previous image corresponding to the previous text into the autoregressive module in the image generation model to obtain the current token sequence output by the autoregressive module; The input module is further used to input the previous image into the image feature adaptation module in the image generation model to obtain the image features output by the image feature adaptation module; A determination module is used to determine a current image corresponding to the current text based on the current token sequence and the image features.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the image generating method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image generating method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image generating method according to any one of claims 1 to 6 is implemented.

Images