Image generation method based on large language model, training method and device of visual large model, and intelligent agent

Abstract

The invention provides an image generation method based on a large language model, a training method and device of a visual large model and an intelligent agent, and relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like. The image generation method based on the large language model comprises the steps of obtaining a demand text; processing the demand text by using a large language model to obtain target demand text features, the target demand text features including image character features and attribute prompt features, the image character features representing image characters needing to be displayed in the generated target image, and the attribute prompt features being related to image attributes of the image characters; and processing the target demand text features by using a visual large model to obtain a target image, the target image comprising an image word region displayed based on image attributes, and the image word region representing image words in the demand text.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, especially to technical fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as AIGC content generation based on artificial intelligence. Background Art

[0002] With the development of artificial intelligence technology, large models are gradually widely used in various fields such as natural language processing, computer vision, speech recognition, etc. For example, the large model can generate corresponding visual images based on natural language descriptions. Summary of the invention

[0003] The present disclosure provides an image generation method based on a large language model, a training method, device, equipment, medium, program product and intelligent agent of a large visual model.

[0004] According to one aspect of the present disclosure, a method for generating an image based on a large language model is provided, comprising: obtaining a demand text; processing the demand text using a large language model to obtain target demand text features, the target demand text features including image word features and attribute prompt features, the image word features representing image words that need to be displayed in a generated target image, the attribute prompt features being related to image attributes of the image words; processing the target demand text features using a large visual model to obtain a target image, the target image including an image word area displayed based on image attributes, the image word area representing image words in the demand text.

[0005] According to another aspect of the present disclosure, a training method for a visual big model is provided, including: obtaining sample target requirement text features, sample noise images and label images, wherein the sample target requirement text features are determined by processing the sample requirement text using a big language model, the sample target requirement text features include sample image word features and sample attribute prompt features, the sample image word features represent label image words displayed in the label image, the attribute prompt features are related to label image attributes of the label image words, and the sample noise image is determined by performing noise processing on the label image; using an initial visual big model to process the sample target requirement text features and the sample noise image features to obtain a sample target image, the sample target image includes a sample image word area related to the label image words, and the sample noise image features are determined based on the sample noise image; and training an initial visual big model based on the sample target image and the label image to obtain a trained visual big model.

[0006] According to another aspect of the present disclosure, there is provided an image generation device based on a large language model, comprising: a first acquisition module, for acquiring a demand text; a first processing module, for processing the demand text using a large language model to obtain target demand text features, wherein the target demand text features include image word features and attribute prompt features, wherein the image word features represent image words that need to be displayed in a generated target image, and wherein the attribute prompt features are related to image attributes of the image words; and a second processing module, for processing the target demand text features using a large visual model to obtain the target image, wherein the target image includes an image word region displayed based on the image attributes, and wherein the image word region represents the image words in the demand text.

[0007] According to another aspect of the present disclosure, a training device for a visual large model is provided, comprising: a second acquisition module, used to acquire sample target requirement text features, a sample noise image and a label image, wherein the sample target requirement text features are determined by processing the sample requirement text using a large language model, the sample target requirement text features include sample image word features and sample attribute prompt features, the sample image word features represent the label image words displayed in the label image, the attribute prompt features are related to the label image attributes of the label image words, and the sample noise image is determined by performing noise processing on the label image; a fourth processing module, used to process the sample target requirement text features and the sample noise image features using the initial visual large model to obtain a sample target image, the sample target image includes a sample image word area related to the label image words, and the sample noise image features are determined based on the sample noise image; and a training module, used to train the initial visual large model based on the sample target image and the label image to obtain a trained visual large model.

[0008] According to another aspect of the present disclosure, there is provided an artificial intelligence agent, comprising: an input module for receiving input information; a processing module for determining a target task based on the input information received by the input module, determining a large language model or a large visual model or an initial large visual model based on the target task, and obtaining output information by calling the large language model and / or the large visual model to execute the above method, or by calling the initial large visual model to execute the above method; and an output module for outputting the output information obtained by the processing module.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0011] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described above when executed by a processor.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 An exemplary system architecture to which a large language model-based image generation method and device, and a large visual model training method and device according to an embodiment of the present disclosure can be applied is schematically shown;

[0015] Figure 2 A flowchart of an image generation method based on a large language model according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 3A Schematically illustrates an embodiment and a comparative example of a target image according to an embodiment of the present disclosure;

[0017] Figure 3B Schematically illustrates an embodiment and a comparative example of an icon image according to another embodiment of the present disclosure;

[0018] Figure 4 A schematic diagram of a visual macro model according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 5 A schematic diagram of a visual macro model according to another embodiment of the present disclosure is schematically shown;

[0020] Figure 6 The flowchart of the method for training a large visual model according to an embodiment of the present disclosure is schematically shown.

[0021] Fig. 7A The label image according to one embodiment of the present disclosure is exemplarily shown;

[0022] Figure 7B exemplarily shows a label image according to another embodiment of the present disclosure;

[0023] Fig. 8A An initial visual macro model according to an embodiment of the present disclosure is exemplarily shown;

[0024] Figure 8B An initial visual macro model according to another embodiment of the present disclosure is exemplarily shown;

[0025] Fig. 9 The output effects of skipping the 1st, 2nd, 10th and 20th layers in the visual large model according to the embodiments of the present disclosure are exemplified;

[0026] Fig.10 The trained visual large model and the pruned visual large model according to an embodiment of the present disclosure are schematically shown.

[0027] Fig.11 A block diagram of an image generation device based on a large language model according to an embodiment of the present disclosure is schematically shown;

[0028] Fig.12 A block diagram of a training device for a large visual model according to an embodiment of the present disclosure is schematically shown;

[0029] Fig.13 A block diagram schematically shows a structure of an artificial intelligence agent according to an embodiment of the present disclosure; and

[0030] Fig.14 A block diagram of an electronic device suitable for implementing an image generation method based on a large language model and a training method of a large visual model according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] In the technical solution disclosed in the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and do not violate public order and good morals.

[0033] With the development of artificial intelligence technology, large models can be applied to the field of computer vision. For example, large models can generate corresponding visual images based on natural language descriptions. However, the inventors found that large models have poor generation effects on the text content required to be displayed in the generated images, and it is difficult to generate appropriate and semantically correct text content images.

[0034] In this regard, the embodiments of the present disclosure provide an image generation method based on a large language model, a training method, device, equipment, medium, program product and intelligent agent for a large visual model. The image generation method based on a large language model includes: obtaining a demand text; processing the demand text using a large language model to obtain target demand text features, the target demand text features include image word features and attribute prompt features, the image word features represent the image words that need to be displayed in the generated target image, and the attribute prompt features are related to the image attributes of the image words; processing the target demand text features using a large visual model to obtain a target image, the target image includes an image word area displayed based on image attributes, and the image word area represents the image words in the demand text.

[0035] According to the embodiments of the present disclosure, by processing the demand text with a large language model, more comprehensive and accurate target demand text features can be obtained to better characterize the image words and image attributes related to the image words that need to be displayed in the target image. Among them, the attribute prompt feature can more accurately describe the image words that need to be displayed in the target image to help the visual large model improve its semantic understanding ability. Based on this, the target demand text features can be used to more accurately control the visual large model to understand the image words that need to be displayed in the target image in the demand text, and then the visual large model can be controlled based on the attribute prompt features to generate the target image based on the image attributes of the image words, so as to improve the accuracy of the image words that need to be displayed in the target image, avoid hallucinations when the visual large model processes the demand text, and make the display effect of the image word area match the image generation requirements of the demand text, thereby improving the overall imaging effect and imaging accuracy of the target image.

[0036] According to the embodiments of the present disclosure, a large model may be a model having a large-scale parameter structure, and the parameter magnitude is generally above 10 million or 100 million, and may reach 10 billion or 10 billion. In terms of the network structure of the large model, for example, a network structure such as UFO (Unified Feature Optimization) may be adopted.

[0037] Figure 1 An exemplary system architecture to which a large language model-based image generation method and apparatus, and a large visual model training method and apparatus can be applied according to an embodiment of the present disclosure is schematically shown.

[0038] It should be noted that Figure 1The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the image generation method and apparatus based on a large language model and the training method and apparatus of a large visual model can be applied may include a terminal device, but the terminal device may implement the image generation method and apparatus based on a large language model and the training method and apparatus of a large visual model provided in the embodiments of the present disclosure without interacting with a server.

[0039] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0040] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0041] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0042] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for the content browsed by the user using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.

[0043] The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server can also be a server of a distributed system, or a server combined with blockchain.

[0044] It should be noted that the image generation method based on the large language model and the training method of the large visual model provided in the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the image generation device based on the large language model and the training device of the large visual model provided in the embodiments of the present disclosure can generally be set in the server 105. The image generation method based on the large language model and the training method of the large visual model provided in the embodiments of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the image generation device based on the large language model and the training device of the large visual model provided in the embodiments of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0045] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0046] Figure 2 The flowchart of the image generation method based on the large language model according to the embodiment of the present disclosure is schematically shown.

[0047] like Figure 2 As shown, the method 200 includes operations S210 to S230.

[0048] In operation S210, a requirement text is obtained.

[0049] In operation S220, the demand text is processed using a large language model to obtain target demand text features, which include image word features and attribute prompt features. The image word features represent image words that need to be displayed in the generated target image, and the attribute prompt features are related to image attributes of the image words.

[0050] In operation S230, the target demand text features are processed using the visual macro model to obtain a target image, wherein the target image includes an image word region displayed based on image attributes, and the image word region represents image words in the demand text.

[0051] According to an embodiment of the present disclosure, the requirement text may include a natural language description or keywords input by a user, and the requirement text may be used to characterize the user's image generation intention, which generally includes the content and features that the user hopes to see in the generated target image. For example, the requirement text may be in a natural language such as Chinese or English, which is not specifically limited here.

[0052] According to an embodiment of the present disclosure, the requirement text includes description information related to the text content that needs to be displayed in the target image. Preferably, the text content can be marked in the requirement text using specific symbols (such as quotation marks, brackets, separators, etc.).

[0053] As an example, the text content may include language characters (such as Chinese and English characters, etc.), special symbols, punctuation marks, Greek characters, Latin characters, mathematical symbols, emoticons, serial numbers, units, etc. For example, the text content may be "ABCD", "tea", "No.1", etc.

[0054] As another example, the text content may also include a logo, icon, trademark, label, special mark, etc. containing the aforementioned text information. For example, the text content may be "XXX icon", "no smoking mark", etc.

[0055] As another example, the text content may also include a logo, icon, trademark, label, special mark, etc. that can be described by the aforementioned text information. For example, the text content may be "A organization logo", "B product trademark", etc.

[0056] For example, the requirement text may be: The image shows a coffee cup in a simple sketch style with the letters "ABCD" on the cup body. Another example is: The image shows a coffee cup in a simple sketch style with the letters "ABCD" on the cup body.

[0057] According to an embodiment of the present disclosure, a large language model can be used to process the demand text to obtain target demand text features, which can include image word features and attribute prompt features. Among them, the image word features represent the image words that need to be displayed in the generated target image, and the attribute prompt features are related to the image attributes of the image words. In one example, the image words can be understood as the text content described above.

[0058] As an example, the image attributes of the image word may include style attributes, texture attributes, size attributes, position attributes, font attributes, etc. related to the above-mentioned text information. As another example, the image attributes of the image word may also include attribute information related to the text content of the above-mentioned preset styles such as LOGO, icon, trademark, label, special mark, etc.

[0059] According to an embodiment of the present disclosure, a large language model (LLM) may be a natural language processing model with large-scale parameters and computing power. Large language models can be widely used in fields such as natural language processing, text generation, and intelligent dialogue. A large language model may be a model with a large-scale parameter structure, and the parameter magnitude is generally above 10 million or 100 million levels, and may reach 1 billion or 10 billion levels. In terms of the network structure of the large language model, for example, a network structure such as a transformer may be used.

[0060] According to an embodiment of the present disclosure, the large language model can process the demand text to obtain image word features and attribute prompt features. Among them, the attribute prompt feature can more accurately describe the image words that need to be displayed in the target image to help the visual large model improve its semantic understanding ability. Based on this, the target demand text features can be used to more accurately control the visual large model to understand the image words in the demand text that need to be displayed in the target image, and then the attribute prompt feature can be used to control the visual large model to generate the target image based on the image attributes of the image words, so as to improve the accuracy of the image words that need to be displayed in the target image, avoid hallucinations when the visual large model processes the demand text, and make the display effect of the image word area match the image generation requirements of the demand text, thereby improving the overall imaging effect and imaging accuracy of the target image.

[0061] According to an embodiment of the present disclosure, a target demand text feature can be processed using a visual macro model to obtain a target image, which includes an image word region displayed based on image attributes, and the image word region represents the image words in the demand text.

[0062] According to one embodiment of the present disclosure, the image words displayed in the target image correspond to the text content contained in the demand text. For example, the image word area may include an area representing the text information in the target image, and the text information may include language characters (such as Chinese and English characters, etc.), special symbols, punctuation marks, Greek characters, Latin characters, mathematical symbols, emoticons, serial numbers, units, etc. For another example, the image word area may also include an area representing the text content in the target image, and the text content may include a LOGO, icon, trademark, label, special mark, etc. that contains the aforementioned text information or can be described by the aforementioned text information.

[0063] According to an embodiment of the present disclosure, a vision big model (Vision Transformer) may be a visual processing model with large-scale parameters and computing power, and the vision big model may be able to extract useful features and information from visual data such as images and videos. The vision big model may be applied to technical fields such as image recognition, target detection, semantic segmentation, and image generation. The vision big model may be a model with a model structure having large-scale parameters, and the magnitude of the parameters is generally in the tens of millions or hundreds of millions, and may reach billions or tens of billions. In terms of the network structure of the vision big model model, for example, network structures such as transformers, convolutional neural networks (CNNs), and dual encoders may be used.

[0064] According to the embodiments of the present disclosure, by processing the demand text with a large language model, more comprehensive and accurate target demand text features can be obtained to better characterize the image words and image attributes related to the image words that need to be displayed in the target image. Among them, the attribute prompt feature can more accurately describe the image words that need to be displayed in the target image to help the visual large model improve its semantic understanding ability. Based on this, the target demand text features can be used to more accurately control the visual large model to understand the image words that need to be displayed in the target image in the demand text, and then the visual large model can be controlled based on the attribute prompt features to generate the target image based on the image attributes of the image words, so as to improve the accuracy of the image words that need to be displayed in the target image, avoid hallucinations when the visual large model processes the demand text, and make the display effect of the image word area match the image generation requirements of the demand text, thereby improving the overall imaging effect and imaging accuracy of the target image.

[0065] Figure 3A An embodiment and a comparative example of a target image according to an embodiment of the present disclosure are exemplarily shown.

[0066] According to an embodiment of the present disclosure, the demand text may be: The image shows a coffee cup in a simple sketch style with the letters "ABCD" on the cup body. Alternatively, the demand text may be: The image shows a coffee cup in a simple sketch style with the letters "ABCD" on the cup body.

[0067] exist Figure 3A In the figure, target image 301 is a target image comparison ratio generated by a related method, and target image 302 is a target image embodiment generated by an image generation method based on a large language model (hereinafter referred to as this method) provided by an embodiment of the present disclosure.

[0068] Figure 3BAn embodiment and a comparative example of an icon image according to another embodiment of the present disclosure are exemplarily shown.

[0069] According to another embodiment of the present disclosure, the demand text may be: The image shows a straw cup in a simple sketch style, with the "XXX Product Logo" displayed at the center of the cup body. Alternatively, the demand text may be: The image shows a straw cup in a simple sketch style, with the "XXX Product Logo" displayed at the center of the cup body.

[0070] exist Figure 3B In the figure, target image 303 is a target image comparison example generated by a related method, and target image 304 is a target image embodiment generated by this method.

[0071] like Figure 3A and Figure 3B As shown, compared with the target image comparison ratio generated by the related method, in the target image embodiment generated by the present method, the generation effect of the text content required to be displayed in the target image is better, and the present method can better generate a suitable and semantically correct text content image.

[0072] According to an embodiment of the present disclosure, the visual big model includes N cascaded image denoising networks, N>1, and N is an integer; wherein, using the visual big model to process the target required text features to obtain the target image includes: based on the attention mechanism, using the cascaded N image denoising networks to process the target required text features and preset noise features to obtain the target image.

[0073] According to an embodiment of the present disclosure, the visual model can be selected as a diffusion model. The diffusion model can be used for image generation, and its inspiration comes from thermodynamics: a distribution can be transformed into another distribution by continuously adding noise. In the image generation task, the images from the training set can be transformed into images that conform to the standard normal distribution by continuously adding noise.

[0074] Exemplarily, the diffusion model consists of a forward process (or diffusion process) and a reverse process (or inverse diffusion process), corresponding to encoding and decoding, respectively. In the forward process, noise is continuously mixed into the input image, and after T noise addition operations, the input image becomes a pure noise image that conforms to the standard normal distribution. In the reverse process, we hope to train a denoising network that can learn T denoising operations and restore the pure noise image back to the input image. Therefore, the learning goal of the denoising network is to allow T denoising operations to just offset the corresponding T noise addition operations. After training, you only need to randomly sample a noise from the standard normal distribution, and then use the denoising network in the reverse process to restore the noise to an image to generate a picture.

[0075] According to an embodiment of the present disclosure, the visual large model can be a diffusion model, and the image denoising network can be used for the reverse process of the diffusion model. The attention mechanism can be used to process the target required text features and the preset noise features by cascading N image denoising networks to obtain the target image. As an example, the preset noise features can be a Gaussian noise matrix.

[0076] According to the embodiments of the present disclosure, different from the encoder and decoder of the traditional diffusion model, the embodiments of the present disclosure adopt the DIT dual-stream architecture (Dual-Stream Diffusion Transformer Architecture) based on the attention mechanism to process the target required text features and noise features. In the DIT dual-stream architecture, two independent streams process image data and text data respectively, and then fuse them at a high level. Since data of different modalities are processed independently, the feature information of each modality can be better preserved. Through independent processing and late fusion, the visual large model can more effectively utilize information of different modalities, as well as better understand and utilize complementary information between different modalities, thereby further improving the semantic understanding ability of the visual large model.

[0077] According to the embodiments of the present disclosure, by utilizing the attention mechanism, the image denoising network can also be helped to better capture the target demand text features and preset noise features globally and locally, and provide a more comprehensive feature representation. Compared with a single-layer image denoising network, a multi-layer cascaded network structure can effectively enhance the model's feature representation capabilities, thereby helping to further improve the semantic understanding capabilities of the large visual model, so as to improve the accuracy of the image words that need to be displayed in the target image, avoid hallucinations in the large visual model, and match the display effect of the image word area with the image generation requirements of the demand text, thereby improving the overall imaging effect and imaging accuracy of the target image.

[0078] According to an embodiment of the present disclosure, an attention mechanism can be used to process target required text features and preset noise features using cascaded N image denoising networks to obtain a target image. As just one example, Latent Flow can be used to process noise features, and Text Flow can be used to process target required text features.

[0079] According to an embodiment of the present disclosure, based on the attention mechanism, N cascaded image denoising networks in the visual big model are used to process the target required text features and the preset noise features to obtain the target image, including: based on the attention mechanism, the nth image denoising network is used to process the n-1th level intermediate text fusion features and the n-1th level intermediate image fusion features to obtain the nth level intermediate text fusion features and the nth level intermediate image fusion features, wherein N≥n>1, and n is an integer; and in the case of N=n, the target image is determined based on the Nth level intermediate image fusion features.

[0080] Figure 4 A schematic diagram of a visual macro model according to an embodiment of the present disclosure is schematically shown.

[0081] According to one embodiment of the present disclosure, Figure 4 As shown, the visual model 400 may include two cascaded image denoising networks, namely a first image denoising network 410 and a second image denoising network 420.

[0082] According to one embodiment of the present disclosure, Figure 4 As shown, based on the attention mechanism, the target required text features and the preset noise features are processed by using the cascaded two image denoising networks in the visual large model 400, and the target image obtained may include:

[0083] For example, when n=2, based on the attention mechanism, the second image denoising network 420 is used to process the first-level intermediate text fusion feature 403 and the first-level intermediate image fusion feature 404 to obtain the second-level intermediate text fusion feature 405 and the second-level intermediate image fusion feature 406.

[0084] For example, when n=2, the target image 407 is determined based on the second-level intermediate image fusion features 406 .

[0085] According to an embodiment of the present disclosure, based on the attention mechanism, the n-1th level intermediate text fusion feature and the n-1th level intermediate image fusion feature are processed by the n-th image denoising network to obtain the n-1th level intermediate text fusion feature and the n-1th level intermediate image fusion feature, which includes: performing spatial domain modulation processing on the n-1th level intermediate text fusion feature and the noise feature based on preset noise information to obtain the n-th spatial domain text feature and the n-th spatial domain image feature corresponding to the preset spatial domain; fusing the n-th spatial domain text feature and the n-th spatial domain image feature based on the attention mechanism to obtain the n-th spatial domain text fusion feature and the n-th spatial domain image fusion feature; and performing spatial domain modulation on the n-th spatial domain text fusion feature and the n-th spatial domain image fusion feature based on the preset noise information to obtain the n-th level intermediate text fusion feature and the n-th level intermediate image fusion feature output by the n-th image denoising network.

[0086] According to an embodiment of the present disclosure, in a dual-stream DiT architecture, spatial modulation processing (Spatial Modulation) can be understood as modulating features in a spatial dimension.

[0087] According to one embodiment of the present disclosure, the visual big model may include three cascaded image denoising networks, namely a first image denoising network, a second image denoising network and a third image denoising network.

[0088] For example, when n=2, the first-level intermediate text fusion features and noise features can be spatially modulated based on preset noise information to obtain second-level spatial domain text features and second-level spatial domain image features corresponding to the preset spatial domain. The second-level spatial domain text features and the second-level spatial domain image features can be fused based on the attention mechanism to obtain second-level spatial domain text fusion features and second-level spatial domain image fusion features. The second-level spatial domain text fusion features and the second-level spatial domain image fusion features can be spatially modulated based on preset noise information to obtain second-level intermediate text fusion features and second-level intermediate image fusion features output by the second image denoising network.

[0089] For example, when n=3, the second-level intermediate text fusion features and noise features can be spatially modulated based on preset noise information to obtain third-level spatial domain text features and third-level spatial domain image features corresponding to the preset spatial domain. The third-level spatial domain text features and the third-level spatial domain image features can be fused based on the attention mechanism to obtain third-level spatial domain text fusion features and third-level spatial domain image fusion features. The third-level spatial domain text fusion features and the third-level spatial domain image fusion features can be spatially modulated based on preset noise information to obtain third-level intermediate text fusion features and third-level intermediate image fusion features output by the third image denoising network.

[0090] According to one embodiment of the present disclosure, the preset noise information may correspond to a preset noise level. Exemplarily, the degree of noise that needs to be added to the n-1th level intermediate text fusion feature and noise feature may be determined based on the noise level. Optionally, the number of loop processing times (e.g., the number of loop processing times is d) for the N image denoising networks cascaded in the visual large model to process the n-1th level intermediate text fusion feature and noise feature may be determined based on the noise level (e.g., the noise level is t).

[0091] In one example, the preset noise information may include 10 preset noise information corresponding to the noise level t=10, and the number of loop processing times for processing the n-1th level intermediate text fusion feature and the noise feature may be, for example, 10. During the first processing, the noise level t may be set to 10, indicating that the noise level added during the first processing is high. As the number of loop processing times increases, the noise level decreases accordingly. During the 10th processing, the noise level t may be set to 1, indicating that the noise level added during the last processing is low.

[0092] It should be understood that the first cycle processing can be understood as using the N image denoising networks of the large visual model to process the target required text features, the preset noise features and the preset noise information to obtain the first round of image fusion features. The second cycle processing can be understood as using the N image denoising networks of the large visual model to process the first round of image fusion features, text fusion features and preset noise information to obtain the second round of image fusion features and text fusion features.

[0093] According to an embodiment of the present disclosure, the first-level intermediate text fusion feature and the first-level intermediate image fusion feature are determined by the first image denoising network processing the target required text feature and the noise feature.

[0094] According to an embodiment of the present disclosure, the first image denoising subnetwork may be used to process target required text features and noise features to determine the first-level intermediate text fusion features and the first-level intermediate image fusion features.

[0095] According to an embodiment of the present disclosure, the target required text features and noise features can be spatially modulated based on preset noise information to obtain the first spatial domain text features and the first spatial domain image features corresponding to the preset spatial domain. The first spatial domain text features and the first spatial domain image features can be fused based on the attention mechanism to obtain the first spatial domain text fusion features and the first spatial domain image fusion features. The first spatial domain text fusion features and the first spatial domain image fusion features can be spatially modulated based on the preset noise information to obtain the first level intermediate text fusion features and the first level intermediate image fusion features output by the first image denoising network.

[0096] According to an embodiment of the present disclosure, the preset noise information may correspond to a preset noise level. Exemplarily, the degree of noise to be added to the target required text feature and the noise feature may be determined according to the noise level. Optionally, the number of loop processing times (e.g., the number of loop processing times is d) of the N image denoising networks cascaded in the visual large model to process the target required text feature and the preset noise feature may be determined according to the noise level (e.g., the noise level is t).

[0097] In one example, the preset noise information may include 10 preset noise information corresponding to the noise level t=10, and the number of loop processing times for processing the target requirement text feature and the preset noise feature may be, for example, 10. During the first processing, the noise level t may be set to 10, indicating that the noise level added during the first processing is high. As the number of loop processing times increases, the noise level decreases accordingly. During the 10th processing, the noise level t may be set to 1, indicating that the noise level added during the last processing is low.

[0098] Figure 5 A schematic diagram of a visual macro model according to another embodiment of the present disclosure is schematically shown.

[0099] According to one embodiment of the present disclosure, Figure 5 As shown, the visual model 500 may include two cascaded image denoising networks, namely a first image denoising network N510 and a second image denoising network N520.

[0100] According to one embodiment of the present disclosure, Figure 5 As shown, based on the attention mechanism, processing the target required text features and the preset noise features using the cascaded two image denoising networks in the visual large model 500 may include the following operations:

[0101] For example, the target demand text feature 502 may be subjected to spatial domain modulation processing based on the preset noise information 501, so as to process the target demand text feature 502 from the text spatial domain to the preset spatial domain, and obtain the first spatial domain text feature 504. For example, the noise feature 503 may be subjected to spatial domain modulation processing based on the preset noise information 501, so as to process the noise feature 503 from the image spatial domain to the preset spatial domain, and obtain the first spatial domain image feature 505.

[0102] For example, the first spatial domain text feature 504 and the first spatial domain image feature 505 can be fused based on the attention mechanism to obtain the first spatial domain text fusion feature 506 and the first spatial domain image fusion feature 507. Based on the attention mechanism, the first spatial domain text feature 504 and the first spatial domain image feature 505 can be fused to obtain the first spatial domain text fusion feature 506 and the first spatial domain image fusion feature 507, thereby enhancing the image denoising network's ability to capture and understand features of different modalities, thereby helping to improve the performance and effect of the large visual model.

[0103] For example, the first spatial domain text fusion feature 506 and the first spatial domain image fusion feature 507 can be spatially modulated based on the preset noise information 501 to obtain the first level intermediate text fusion feature 508 and the first level intermediate image fusion feature 509 output by the first image denoising network N510.

[0104] For example, the first spatial domain text fusion feature 506 may be spatially modulated based on the preset noise information 501, so as to process the first spatial domain text fusion feature 506 from the preset spatial domain to the text spatial domain, and obtain the first level intermediate text fusion feature 508 output by the first image denoising network N510. For example, the first spatial domain image fusion feature 507 may be spatially modulated based on the preset noise information 501, so as to process the first spatial domain image fusion feature 507 from the preset spatial domain to the image spatial domain, and obtain the first level intermediate image fusion feature 509 output by the first image denoising network N510.

[0105] For example, the first-level intermediate text fusion feature 508 may be subjected to spatial domain modulation processing based on the preset noise information 501, so as to process the first-level intermediate text fusion feature 508 from the text spatial domain to the preset spatial domain, and obtain the second spatial domain text feature 510. For example, the first-level intermediate text fusion feature 508 may be subjected to spatial domain modulation processing based on the preset noise information 501, so as to process the first-level intermediate text fusion feature 508 from the image spatial domain to the preset spatial domain, and obtain the second spatial domain image feature 511.

[0106] For example, the second spatial domain text feature 510 and the second spatial domain image feature 511 can be fused based on the attention mechanism to obtain the second spatial domain text fusion feature 512 and the second spatial domain image fusion feature 513. Based on the attention mechanism, the second spatial domain text feature 510 and the second spatial domain image feature 511 can be fused to obtain the second spatial domain text fusion feature 512 and the second spatial domain image fusion feature 513, thereby enhancing the image denoising network's ability to capture and understand features of different modalities, thereby helping to improve the performance and effect of the large visual model.

[0107] For example, the second spatial domain text fusion feature 512 may be spatially modulated based on the preset noise information 501 to convert the second spatial domain text fusion feature 512 from the preset spatial domain to the text spatial domain, thereby obtaining the second-level intermediate text fusion feature 514 output by the second image denoising network N520. For example, the second spatial domain image fusion feature 513 may be spatially modulated based on the preset noise information 501 to convert the second spatial domain image fusion feature 513 from the preset spatial domain to the image spatial domain, thereby obtaining the second-level intermediate image fusion feature 515 output by the second image denoising network N520.

[0108] For example, the second-level intermediate text fusion feature 514 and the second-level intermediate image fusion feature 515 output by the second image denoising network N520 can be input into the first image denoising network N510 for further processing until it is iterated d times, and finally the visual large model 500 outputs the target image 516. The number of iterations d is determined based on the noise level t represented by the preset noise information 501.

[0109] According to an embodiment of the present disclosure, the image generation method based on the large language model also includes: in response to a text editing request, updating the target requirement text to obtain an updated target requirement text, wherein the target requirement text is determined based on the target requirement text features; and using the visual large model to process the updated target requirement text features to obtain an updated target image, wherein the updated target requirement text features are determined based on the updated target requirement text.

[0110] According to an embodiment of the present disclosure, the generated target image can be fed back to the user. The user can continue to update the required text to adjust the target image and trigger a text editing request.

[0111] According to an embodiment of the present disclosure, in response to a text editing request, the updated requirement text can be processed using a large language model to update the target requirement text and obtain an updated target requirement text. The target requirement text is determined based on the target requirement text features. For example, the target requirement text can be a prompt that is more accurate and contains descriptive information related to the text content that needs to be displayed in the target image, obtained by the large language model based on the target requirement text features. The updated target requirement text features can be determined based on the updated target requirement text. The updated target requirement text features can be processed using a large visual model to obtain an updated target image.

[0112] According to the embodiments of the present disclosure, by updating the target requirement text, the target requirement text can be further adjusted based on user feedback to improve the accuracy of the target requirement text features, so that the visual big model can be more accurately controlled to understand the image words in the requirement text that need to be displayed in the target image, and then the visual big model can be controlled based on the attribute prompt features to generate the target image based on the image attributes to improve the accuracy of the image words that need to be displayed in the target image, avoid hallucinations when the visual big model processes the requirement text, and make the display effect of the image word area match the image generation requirements of the requirement text, thereby improving the overall imaging effect and imaging accuracy of the target image.

[0113] According to an embodiment of the present disclosure, the image attribute includes at least one of the following: a style attribute, a texture attribute, a font attribute, and a preset style attribute.

[0114] As an example, the font attribute may represent the font size, font, font style, color, etc. of the text information. The preset style attribute may represent the attribute information related to the text content of the preset style such as LOGO, icon, trademark, label, special mark, etc.

[0115] Figure 6 The flowchart of the method for training a large visual model according to an embodiment of the present disclosure is schematically shown.

[0116] like Figure 6 As shown, the method 600 includes operations S610-S620.

[0117] In operation S610, sample target requirement text features, sample noise images and label images are obtained. The sample target requirement text features are determined by processing the sample requirement text using a large language model. The sample target requirement text features include sample image word features and sample attribute prompt features. The sample image word features represent label image words displayed in the label image. The attribute prompt features are related to label image attributes of the label image words. The sample noise image is determined by performing noise processing on the label image.

[0118] In operation 620, the sample target requirement text features and the sample noise image features are processed using the initial visual large model to obtain a sample target image, where the sample target image includes a sample image word region associated with the label image word, and the sample noise image features are determined based on the sample noise image.

[0119] In operation 630 , an initial visual large model is trained based on the sample target image and the label image to obtain a trained visual large model.

[0120] According to an embodiment of the present disclosure, a label image contains label image characters, and a sample noise image can be obtained by performing noise processing on the label image. Exemplarily, the sample noise image can be a Gaussian noise image.

[0121] According to an embodiment of the present disclosure, sample demand texts can be processed based on a large language to obtain sample target demand text features. Sample target demand text features can include sample image word features and sample attribute prompt features. Among them, the sample image word features represent the label image words displayed in the label image, and the attribute prompt features are related to the label image attributes of the label image words.

[0122] As an example, the label image attributes of the label image word may include style attributes, texture attributes, size attributes, position attributes, font attributes, etc. related to the text information described above. As another example, the label image attributes of the label image word may also include attribute information related to the text content of the preset style such as the LOGO, icon, trademark, label, special mark, etc. described above.

[0123] According to an embodiment of the present disclosure, an initial visual big model may be trained based on a sample target image and a label image to obtain a trained visual big model.

[0124] According to an embodiment of the present disclosure, a sample set may be constructed based on a label image and a label text. Exemplarily, the sample set may include a first sample set and a second sample set.

[0125] As an example, the first sample set may be constructed as follows:

[0126] a. Directed collection of label images with text information, where the text information can be understood as the above-mentioned label image characters. For example, the text information may include language characters (such as Chinese and English characters, etc.), special symbols, punctuation marks, Greek characters, Latin characters, mathematical symbols, emoticons, serial numbers, units, etc.;

[0127] b. Recognizing text information in the label image based on, for example, an optical character recognition algorithm;

[0128] c. Use the large language model to process the recognized text information and the image description of the label image to obtain the label text embedded with the text information, and mark the text information in the label text with specific symbols (such as quotation marks).

[0129] Fig. 7A A label image according to an embodiment of the present disclosure is exemplarily shown.

[0130] According to an embodiment of the present disclosure, the label text may be: In this cartoon-style sticker picture, the light beige background gently wraps every detail. At the bottom of the picture, a line of eye-catching red capital letters "GO AWAY IM READING" is particularly eye-catching. This line of text not only forms a sharp contrast with the background, but also conveys an atmosphere of tranquility and concentration, as if telling the outside world that the child is enjoying his own reading time at this moment and does not want to be disturbed.

[0131] As another example, the second sample set may be constructed as follows:

[0132] a. Directed collection and display of label images with text content, where the text content can be understood as the above-mentioned label image words. For example, the text content may include LOGOs, icons, trademarks, labels, special marks, etc. that contain the above-mentioned text information or can be described by the above-mentioned text information;

[0133] b. Recognize the text content in the collected label images based on the preset list and recognition operator to obtain text content description information;

[0134] c. For labeled images with unrecognized text content, use the retrieval system to collect description information and manually annotate the text content description information;

[0135] d. Use the large language model to process the recognized text content description information and the image description about the label image to obtain the label text embedded with the text content description information, and mark the text content description information in the label text with specific symbols (such as quotation marks).

[0136] Figure 7B A label image according to another embodiment of the present disclosure is exemplarily shown.

[0137] According to an embodiment of the present disclosure, for example, the sample text may be: a white yogurt bottle marked with "XYZ", with a simple bottle design and gold and red decorative patterns and text descriptions on the front. The main pattern is cherry and rose elements, and the text description includes information such as flavor and net content. The bottom of the bottle looks wider, the upper part is bottleneck-shaped, and the bottle cap is made of white plastic. The logo "XYZ" is located in the center of the picture.

[0138] According to an embodiment of the present disclosure, the initial visual big model includes N cascaded initial image denoising networks, N>1, and N is an integer; wherein, using the initial visual big model to process the target requirement text features and the sample noise image features to obtain the sample target image includes: based on the attention mechanism, using the cascaded N initial image denoising networks to process the sample target requirement text features and the sample noise features to obtain the sample target image.

[0139] According to an embodiment of the present disclosure, the initial visual model may be a diffusion model, and the initial image denoising network may be used for the reverse process of the diffusion model. The attention mechanism may be used to process the sample target requirement text features and the sample noise features using the cascaded N initial image denoising networks to obtain the sample target image. As an example, the sample noise features may be a Gaussian noise matrix.

[0140] According to the embodiments of the present disclosure, the inventors have found that increasing the number of parameters of the large visual model has a positive effect on the generation effect of the target image. However, the relevant large visual model is often trained from scratch after the number of parameters is expanded, which has the problem of long training time.

[0141] According to an embodiment of the present disclosure, the initial visual large model is determined based on the following operations: obtaining a pre-trained visual model, wherein the pre-trained visual model includes K pre-trained image denoising networks, K is an integer greater than 1; replicating the network structure of at least one pre-trained image denoising network in the pre-trained visual model to obtain an extended image denoising network; and updating the pre-trained visual model based on at least one extended image denoising network to obtain an initial visual large model with N initial image denoising networks, N>K.

[0142] According to the embodiments of the present disclosure, it can be understood that the pre-trained image denoising network and the extended image denoising network are both pre-trained networks, which have a certain generalization ability. Therefore, in subsequent model training tasks, the model can converge faster, thereby accelerating the training process.

[0143] According to the embodiments of the present disclosure, by using the pre-trained denoising network and the extended image denoising network, the parameter amount of the initial large visual model can be quickly and conveniently expanded, which helps to improve the performance of the large visual model in generating target images. By updating the pre-trained visual model based on the extended image denoising network, the amount of parameters that need to be retrained can be reduced while ensuring the total number of model parameters, thereby effectively reducing the time required for training and reducing the consumption of computing resources.

[0144] According to an embodiment of the present disclosure, the i-th initial image denoising network of the initial visual large model is the j-th pre-trained image denoising network of the pre-trained visual model, and the i+1-th initial image denoising network of the initial visual large model is an extended image denoising network corresponding to the j-th pre-trained image denoising network, where i and j are integers.

[0145] Fig. 8A An initial visual macro model according to an embodiment of the present disclosure is exemplarily shown.

[0146] According to one embodiment of the present disclosure, Fig. 8AAs shown, the pre-trained visual model may include three pre-trained image denoising networks, namely layer1, layer2 and layer3. The network structure of the three pre-trained image denoising networks in the pre-trained visual model may be replicated to obtain three extended image denoising networks, namely layer1', layer2' and layer3'. The initial visual large model 810 may be obtained by stacking in the pattern of layer1, layer1', layer2, layer2', layer3, layer3'.

[0147] exist Fig. 8A For example, the third initial image denoising network of the initial visual large model 810 is the first pre-trained image denoising network layer1 of the pre-trained visual model, and the fourth initial image denoising network of the initial visual large model is the extended image denoising network layer1' corresponding to the first pre-trained image denoising network layer1.

[0148] According to an embodiment of the present disclosure, the initial visual large model includes a cascaded i-th initial image denoising module and an i+1-th initial image denoising module; wherein, the multiple initial image denoising networks in the i-th initial image denoising module are multiple target pre-trained image denoising networks of the pre-trained visual model, and the multiple initial image denoising networks in the i+1-th initial image denoising module are multiple associated extended image denoising networks; the multiple associated extended image denoising networks are determined by replicating the network structure of multiple target pre-trained image denoising networks.

[0149] Figure 8B An initial visual macro model according to another embodiment of the present disclosure is exemplarily shown.

[0150] According to one embodiment of the present disclosure, Figure 8B As shown, the pre-trained visual model may include three pre-trained image denoising networks, namely layer1, layer2 and layer3. The network structure of the three target pre-trained image denoising networks may be replicated to obtain three extended image denoising networks, namely layer1', layer2' and layer3'. The initial visual large model 800 may be obtained by stacking in the pattern of layer1, layer2, layer3, layer1', layer2', layer3'.

[0151] exist Figure 8BIn the figure, the initial visual model 8200 includes a cascaded first initial image denoising module 821 and a second initial image denoising module 822, layer1, layer2 and layer3 constitute the first initial image denoising module 821, and layer1', layer2' and layer3' constitute the second initial image denoising module 822.

[0152] According to the embodiments of the present disclosure, the inventors also found that the large visual model with a large number of parameters performs poorly under limited computing resources, and there is a problem of too long target image generation time. Furthermore, the inventors found that by pruning the parameters of the large visual model, the image generation effect can be reduced less while effectively improving the image generation speed.

[0153] According to an embodiment of the present disclosure, the inventor observes the influence of each layer of the visual large model on the overall image output effect by skipping the image output effect after a certain layer in the visual large model when outputting the target image.

[0154] Fig. 9 The following example shows the output effect of skipping the 1st, 2nd, 10th and 20th layers in the visual model according to the embodiment of the present disclosure. For example, the prompt is: three medical personnel, female, 18 years old, with short black hair, smiling, standing in a dignified posture, and the background is a hospital. Fig. 9 It can be observed that skipping the 10th and 20th layers has little impact on the overall output effect.

[0155] According to an embodiment of the present disclosure, the visual big model includes N cascaded image denoising networks, N>1, and N is an integer; the training method of the visual big model also includes: pruning the trained visual big model to obtain a pruned visual big model; and performing knowledge distillation on the pruned visual big model according to the trained visual big model to obtain an updated visual big model.

[0156] According to the embodiments of the present disclosure, by pruning the trained visual big model, the number of parameters of the visual big model can be effectively reduced, which helps to reduce the energy consumption of the model and improve the image generation speed of the model. By performing knowledge distillation on the pruned visual big model according to the trained visual big model, the knowledge of the visual big model with a large number of parameters can be transferred to the pruned visual big model with a small number of parameters, so that the pruned visual big model can maintain or approach the accuracy and performance of the original model while being lightweight, ensuring that the pruned visual big model can have a better target image generation effect.

[0157] According to an embodiment of the present disclosure, the pruned visual big model includes M cascaded pruned image denoising networks, N>M>1, and M is an integer; wherein, performing knowledge distillation on the pruned visual big model according to the trained visual big model includes: using the trained visual big model to process sample scene requirement text features and sample scene noise image features related to the specified scene, and obtaining the Nth level intermediate scene image fusion features and the Nth level intermediate scene text fusion features output by the Nth image denoising network; using the pruned visual big model to process the sample scene requirement text features and the sample scene noise image features, and obtaining the Mth level intermediate pruned image fusion features and the Mth level intermediate pruned text fusion features output by the Mth pruned image denoising network; determining the knowledge distillation loss based on the Nth level intermediate scene image fusion features and the Nth level intermediate scene text fusion features, as well as the Mth level intermediate pruned image fusion features and the Mth level intermediate pruned text fusion features; and adjusting the model parameters of the Mth pruned image denoising network according to the knowledge distillation loss.

[0158] According to an embodiment of the present disclosure, the sample scene requirement text feature and the sample scene noise image feature are determined based on the sample scene requirement text and the sample scene image related to the specified scene.

[0159] Fig.10 The trained visual large model and the pruned visual large model according to the embodiments of the present disclosure are exemplarily shown.

[0160] According to one embodiment of the present disclosure, Fig.10 As shown, the trained visual model 1010 may include 20 cascaded image denoising networks (block1 to block20). The trained visual model 1010 may be pruned to obtain a pruned visual model 1020. Blocks 10 to 20 in the trained visual model 1010 are removed to obtain a pruned visual model 1020. The pruned visual model 1020 may include 10 cascaded image denoising networks (block1 to block10).

[0161] As an example, Fig.10 As shown, the trained visual macro model 1010 can be used to process the sample scene requirement text feature 1001 and the sample scene noise image feature 1002 related to the specified scene, and obtain the 20th level intermediate scene text fusion feature 1003 and the 20th level intermediate scene image fusion feature 1004 output by the 20th image denoising network (block20). Among them, the sample scene requirement text feature 1001 and the sample scene noise image feature 1002 are determined based on the sample scene requirement text and the sample scene image related to the specified scene.

[0162] As an example, Fig.10As shown, the pruned visual large model 1020 can be used to process the sample scene required text features 1001 and the sample scene noise image features 1002 to obtain the 10th level intermediate pruned text fusion features 1005 and the 10th level intermediate pruned image fusion features 1006 output by the 10th pruned image denoising network (block10).

[0163] As an example, Fig.10 As shown, the knowledge distillation loss can be determined based on the 20th level intermediate scene text fusion feature 1003 and the 20th level intermediate scene image fusion feature 1004, and the 10th level intermediate pruning text fusion feature 1005 and the 10th level intermediate pruning image fusion feature 1006. For example, the knowledge distillation text loss 1007 can be determined based on the 20th level intermediate scene text fusion feature 1003 and the 10th level intermediate pruning text fusion feature 1005. The knowledge distillation image loss 1008 can be determined based on the 20th level intermediate scene image fusion feature 1004 and the 10th level intermediate pruning image fusion feature 1006. The knowledge distillation loss 1009 can be determined based on the knowledge distillation image loss and the knowledge distillation text loss.

[0164] As an example, Fig.10 As shown, the model parameters of the 10th pruned image denoising network (block10) can be adjusted according to the knowledge distillation loss 1009.

[0165] Fig.11 A block diagram of an image generation device based on a large language model according to an embodiment of the present disclosure is schematically shown.

[0166] like Fig.11 As shown, the image generation device 1100 based on the large language model may include a first acquisition module 1110 , a first processing module 1120 , and a second processing module 1130 .

[0167] The first acquisition module 1110 is used to acquire the requirement text.

[0168] The first processing module 1120 is used to process the demand text using a large language model to obtain target demand text features, wherein the target demand text features include image word features and attribute prompt features, wherein the image word features represent image words that need to be displayed in the generated target image, and the attribute prompt features are related to the image attributes of the image words.

[0169] The second processing module 1130 is used to process the target demand text features using the visual big model to obtain the target image, wherein the target image includes an image word region displayed based on the image attributes, and the image word region represents the image words in the demand text.

[0170] According to an embodiment of the present disclosure, the visual large model includes N cascaded image denoising networks, N>1, and N is an integer; wherein the second processing module includes a first image denoising processing submodule.

[0171] The first image denoising processing submodule is used to process target required text features and preset noise features based on the attention mechanism by utilizing cascaded N image denoising networks to obtain a target image.

[0172] According to an embodiment of the present disclosure, the first image noise reduction processing submodule includes a first processing unit and a second processing unit.

[0173] The first processing unit is used to process the n-1th level intermediate text fusion features and the n-1th level intermediate image fusion features based on the attention mechanism using the nth image denoising network to obtain the nth level intermediate text fusion features and the nth level intermediate image fusion features, wherein N≥n>1, and n is an integer, and the first level intermediate text fusion features and the first level intermediate image fusion features are determined by the target text features and noise features processed by the first image denoising network.

[0174] The second processing unit is used to determine the target image based on the Nth level intermediate image fusion feature when N=n.

[0175] According to an embodiment of the present disclosure, the first processing unit may include a first spatial domain modulation subunit, a feature fusion subunit and a second spatial domain modulation subunit.

[0176] The first spatial domain modulation subunit is used to perform spatial domain modulation processing on the n-1th level intermediate text fusion feature and the noise feature based on preset noise information to obtain the nth spatial domain text feature and the nth spatial domain image feature corresponding to the preset spatial domain.

[0177] The feature fusion subunit is used to fuse the nth spatial domain text feature and the nth spatial domain image feature based on the attention mechanism to obtain the nth spatial domain text fusion feature and the nth spatial domain image fusion feature.

[0178] The second spatial domain modulation subunit is used to perform spatial domain modulation on the nth spatial domain text fusion feature and the nth spatial domain image fusion feature based on preset noise information to obtain the nth level intermediate text fusion feature and the nth level intermediate image fusion feature output by the nth image denoising network.

[0179] According to an embodiment of the present disclosure, the first-level intermediate text fusion feature and the first-level intermediate image fusion feature are determined by the first image denoising network processing the target required text feature and the noise feature.

[0180] According to an embodiment of the present disclosure, the image generation device based on the large language model further includes an updating module and a third processing module.

[0181] The updating module is used to update the target requirement text in response to the text editing request to obtain the updated target requirement text, wherein the target requirement text is determined based on the target requirement text features.

[0182] The third processing module is used to process the updated target requirement text features using the visual large model to obtain an updated target image, where the updated target requirement text features are determined based on the updated target requirement text.

[0183] According to an embodiment of the present disclosure, the image attribute includes at least one of the following: a style attribute, a texture attribute, a font attribute, and a preset style attribute.

[0184] Fig.12 A block diagram of a training device for a large visual model according to an embodiment of the present disclosure is schematically shown.

[0185] like Fig.12 As shown, the training device 1200 for the large visual model may include a second acquisition module 1210 , a fourth processing module 1220 and a training module 1230 .

[0186] The second acquisition module 1210 is used to obtain sample target requirement text features, sample noise images and label images. The sample target requirement text features are determined by processing the sample requirement text using a large language model. The sample target requirement text features include sample image word features and sample attribute prompt features. The sample image word features represent the label image words displayed in the label image. The attribute prompt features are related to the label image attributes of the label image words. The sample noise image is determined by performing noise processing on the label image.

[0187] The fourth processing module 1220 is used to use the initial visual large model to process the sample target requirement text features and the sample noise image features to obtain a sample target image, wherein the sample target image includes a sample image word area related to the label image word, and the sample noise image features are determined based on the sample noise image.

[0188] The training module 1230 is used to train the initial visual large model based on the sample target image and the label image to obtain a trained visual large model.

[0189] According to an embodiment of the present disclosure, the initial visual large model includes N cascaded initial image denoising networks, N>1, and N is an integer; wherein the fourth processing module includes a second image denoising processing submodule.

[0190] The second image denoising processing submodule is used to process the sample target required text features and the sample noise features based on the attention mechanism by utilizing the cascaded N initial image denoising networks to obtain the sample target image.

[0191] According to an embodiment of the present disclosure, the initial visual macro model is determined based on the following operations:

[0192] Obtain a pre-trained visual model, wherein the pre-trained visual model includes K pre-trained image denoising networks, where K is an integer greater than 1;

[0193] Copying the network structure of at least one pre-trained image denoising network in the pre-trained visual model to obtain an extended image denoising network; and

[0194] The pre-trained visual model is updated based on at least one extended image denoising network to obtain an initial visual large model with N initial image denoising networks, where N>K.

[0195] According to an embodiment of the present disclosure, the i-th initial image denoising network of the initial visual large model is the j-th pre-trained image denoising network of the pre-trained visual model, and the i+1-th initial image denoising network of the initial visual large model is an extended image denoising network corresponding to the j-th pre-trained image denoising network, where i and j are integers.

[0196] According to an embodiment of the present disclosure, the initial visual large model includes a cascaded i-th initial image denoising module and an i+1-th initial image denoising module; wherein, the multiple initial image denoising networks in the i-th initial image denoising module are multiple pre-trained image denoising networks of a pre-trained visual model, and the multiple initial image denoising networks in the i+1-th initial image denoising module are multiple extended image denoising networks; the multiple extended image denoising networks are determined by replicating the network structure of multiple pre-trained image denoising networks.

[0197] According to an embodiment of the present disclosure, the visual large model includes N cascaded image denoising networks, N>1, and N is an integer; the training device of the visual large model also includes a pruning module and a knowledge distillation module.

[0198] The pruning module is used to prune the trained visual large model to obtain a pruned visual large model.

[0199] The knowledge distillation module is used to perform knowledge distillation on the pruned visual big model according to the trained visual big model to obtain an updated visual big model.

[0200] According to an embodiment of the present disclosure, the pruned visual large model includes M cascaded pruned image denoising networks, N>M>1, and M is an integer; wherein the knowledge distillation module includes a first processing submodule, a second processing submodule, a loss determination submodule and a training submodule.

[0201] The first processing submodule is used to use the trained visual large model to process the sample scene requirement text features and sample scene noise image features related to the specified scene, and obtain the Nth level intermediate scene image fusion features and the Nth level intermediate scene text fusion features output by the Nth image denoising network.

[0202] The second processing submodule is used to use the pruned visual large model to process the sample scene required text features and the sample scene noise image features to obtain the Mth level intermediate pruned image fusion features and the Mth level intermediate pruned text fusion features output by the Mth pruned image denoising network.

[0203] The loss determination submodule is used to determine the knowledge distillation loss based on the N-th level intermediate scene image fusion feature and the N-th level intermediate scene text fusion feature, and the M-th level intermediate pruned image fusion feature and the M-th level intermediate pruned text fusion feature.

[0204] The training submodule is used to adjust the model parameters of the Mth pruned image denoising network according to the knowledge distillation loss.

[0205] Fig.13 The structural block diagram of an artificial intelligence agent according to an embodiment of the present disclosure is schematically shown.

[0206] In the embodiments of the present disclosure, Fig.13 As shown, the AI ​​agent 1300 may include an input module 1310 , a processing module 1320 , and an output module 1330 .

[0207] The input module 1310 is used to receive input information.

[0208] The processing module 1320 is used to determine the target task based on the input information received by the input module, determine the big language model or the big visual model or the initial big visual model based on the target task, and obtain output information by calling the big language model and / or the big visual model to execute the above-mentioned image generation method based on the big language model, or by calling the initial big visual model to execute the above-mentioned training method of the big visual model.

[0209] The output module 1330 is used to output the output information obtained by the processing module.

[0210] According to an embodiment of the present disclosure, the input module 1310 is responsible for receiving or sensing information such as queries, requests, instructions, signals or data from the outside world (e.g., users or external environment), and converting them into a format that can be understood and processed by the AI ​​agent 1300. The input module 1310 is the primary link for the AI ​​agent 1300 to interact with the outside world, which enables the AI ​​agent 1300 to efficiently and accurately obtain the necessary "sensory" information from the outside world and respond to this information.

[0211] In an example, the input module 1310 may input the requirement text or sample target requirement text features, sample noise images, and label images described above.

[0212] In the example, the processing module 1320 is the core support for the AI ​​agent 1300 to process complex tasks. The processing module 1320 can execute the image generation method based on the large language model and the training method of the large visual model described above.

[0213] In the example, the performance of the processing module 1320 may be closely related to the large model on which the AI ​​agent 1300 is based. In order to give full play to the capabilities of the large model, the internal structure of the processing module 1320 may be designed to be highly configurable and extensible so as to cope with various types of tasks and requirements in real scenarios.

[0214] In the example, after the AI ​​agent 1300 obtains the demand text, the processing module 1320 can use the large language model to process the demand text to obtain the target demand text features. The processing module 1320 can use the large visual model to process the target demand text features to obtain the target image, and pass the target image to the output module 1330.

[0215] In the example, after the AI ​​agent 1300 obtains the sample target requirement text features, the sample noise image and the label image, the processing module 1320 can use the initial visual large model to process the sample target requirement text features and the sample noise image features to obtain the sample target image. The processing module 1320 can train the initial visual large model based on the sample target image and the label image to obtain the trained visual large model, and pass the trained visual large model to the output module 1330.

[0216] It is understandable that although the large language model has excellent language understanding and generation capabilities, it is just like a human being and can only solve very limited tasks without any tools. When the AI ​​agent 1300 is given the ability to call tools, it can perform tasks such as using a calculator to perform mathematical operations, using Python to perform data analysis, and using a search engine to perform weather forecasts.

[0217] In an example, the output module 1330 may output the target image or the trained visual large model described above.

[0218] The AI ​​agent 1300 according to the embodiment of the present disclosure can simply and effectively improve the level of intelligence, and improve flexibility and versatility.

[0219] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0220] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0221] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0222] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method as described above.

[0223] Fig.14 A schematic block diagram of an example electronic device 1400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0224] like Fig.14 As shown, the device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1402 or a computer program loaded from a storage unit 1408 into a random access memory (RAM) 1403. In the RAM 1403, various programs and data required for the operation of the device 1400 can also be stored. The computing unit 1401, the ROM 1402, and the RAM 1403 are connected to each other via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0225] A number of components in the device 1400 are connected to the I / O interface 1405, including: an input unit 1406, such as a keyboard, a mouse, etc.; an output unit 1407, such as various types of displays, speakers, etc.; a storage unit 1408, such as a disk, an optical disk, etc.; and a communication unit 1409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1409 allows the device 1400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0226] The computing unit 1401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1401 performs the various methods and processes described above, such as an image generation method based on a large language model and a training method for a large visual model. For example, in some embodiments, the image generation method based on a large language model and the training method for a large visual model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1400 via the ROM 1402 and / or the communication unit 1409. When the computer program is loaded into the RAM 1403 and executed by the computing unit 1401, one or more steps of the image generation method based on the large language model and the training method for the large visual model described above may be performed. Alternatively, in other embodiments, the computing unit 1401 may be configured in any other appropriate manner (eg, by means of firmware) to execute an image generation method based on a large language model or a training method for a large visual model.

[0227] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0228] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0229] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0230] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0231] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0232] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0233] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0234] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating an image based on a large language model, comprising: Get the requirement text; Processing the demand text using a large language model to obtain target demand text features, wherein the target demand text features include image word features and attribute prompt features, wherein the image word features represent image words that need to be displayed in the generated target image, and the attribute prompt features are related to image attributes of the image words; The target demand text features are processed using a visual large model to obtain the target image, wherein the target image includes an image word region displayed based on the image attributes, and the image word region represents the image words in the demand text.

2. The method according to claim 1, wherein: The visual large model includes N cascaded image denoising networks, N>1, and N is an integer; Wherein, the step of processing the target requirement text features using a large visual model to obtain the target image includes: Based on the attention mechanism, the cascaded N image denoising networks are used to process the target required text features and the preset noise features to obtain the target image.

3. The method according to claim 2, wherein: The attention mechanism is based on using the cascaded N image denoising networks in the visual large model to process the target required text features and the preset noise features to obtain the target image, including: Based on the attention mechanism, the nth image denoising network is used to process the n-1th level intermediate text fusion feature and the n-1th level intermediate image fusion feature to obtain the nth level intermediate text fusion feature and the nth level intermediate image fusion feature, where N≥n>1 and n is an integer; and When N=n, the target image is determined based on the Nth level intermediate image fusion features.

4. The method according to claim 3, wherein: Based on the attention mechanism, the n-th image denoising network is used to process the n-1th level intermediate text fusion features and the n-1th level intermediate image fusion features, and the n-th level intermediate text fusion features and the n-th level intermediate image fusion features are obtained, including: Performing spatial domain modulation processing on the n-1th level intermediate text fusion feature and the noise feature based on preset noise information to obtain an nth spatial domain text feature and an nth spatial domain image feature corresponding to the preset spatial domain; fusing the nth spatial domain text feature and the nth spatial domain image feature based on an attention mechanism to obtain an nth spatial domain text fusion feature and an nth spatial domain image fusion feature; and The nth spatial domain text fusion feature and the nth spatial domain image fusion feature are spatially modulated based on preset noise information to obtain the nth level intermediate text fusion feature and the nth level intermediate image fusion feature output by the nth image denoising network.

5. The method according to claim 3 or 4, wherein: The first-level intermediate text fusion features and the first-level intermediate image fusion features are determined by the first image denoising network processing the target required text features and the noise features.

6. The method according to claim 1, wherein: Also includes: In response to the text editing request, the target requirement text is updated to obtain an updated target requirement text, wherein the target requirement text is determined based on the target requirement text feature; as well as The updated target requirement text features are processed using the visual macromodel to obtain an updated target image, wherein the updated target requirement text features are determined based on the updated target requirement text.

7. The method according to claim 1, wherein: The image attributes include at least one of the following: Style attributes, texture attributes, font attributes, preset style attributes.

8. A method for training a large visual model, comprising: Acquire sample target demand text features, sample noise images and label images, wherein the sample target demand text features are determined by processing the sample demand text using a large language model, the sample target demand text features include sample image word features and sample attribute prompt features, the sample image word features represent label image words displayed in the label image, the attribute prompt features are related to label image attributes of the label image words, and the sample noise image is determined by performing noise processing on the label image; Processing the sample target requirement text features and the sample noise image features using the initial visual large model to obtain a sample target image, wherein the sample target image includes a sample image word region associated with the label image word, and the sample noise image features are determined based on the sample noise image; as well as The initial visual large model is trained based on the sample target image and the label image to obtain a trained visual large model.

9. The method according to claim 8, wherein: The initial visual large model includes N cascaded initial image denoising networks, N>1, and N is an integer; The step of using the initial visual large model to process the target requirement text features and the sample noise image features to obtain the sample target image includes: Based on the attention mechanism, the cascaded N initial image denoising networks are used to process the sample target requirement text features and the sample noise features to obtain the sample target image.

10. The method according to claim 8 or 9, wherein: The initial visual macro model is determined based on the following operations: Obtain a pre-trained visual model, wherein the pre-trained visual model includes K pre-trained image denoising networks, where K is an integer greater than 1; Copying the network structure of at least one of the pre-trained image denoising networks in the pre-trained visual model to obtain an extended image denoising network; and The pre-trained visual model is updated based on at least one of the extended image denoising networks to obtain an initial visual large model with N initial image denoising networks, N>K.

11. The method according to claim 10, wherein: The i-th initial image denoising network of the initial visual large model is the j-th pre-trained image denoising network of the pre-trained visual model, and the i+1-th initial image denoising network of the initial visual large model is the extended image denoising network corresponding to the j-th pre-trained image denoising network, where i and j are integers.

12. The method according to claim 10, wherein: The initial visual large model includes a cascaded i-th initial image denoising module and an i+1-th initial image denoising module; Among them, the multiple initial image denoising networks in the i-th initial image denoising module are multiple pre-trained image denoising networks of the pre-trained visual model, and the multiple initial image denoising networks in the i+1-th initial image denoising module are multiple extended image denoising networks; the multiple extended image denoising networks are determined by replicating the network structure of the multiple pre-trained image denoising networks.

13. The method according to claim 8, wherein: The visual large model includes N cascaded image denoising networks, N>1, and N is an integer; The method further comprises: Pruning the trained visual large model to obtain a pruned visual large model; and The pruned visual big model is subjected to knowledge distillation according to the trained visual big model to obtain an updated visual big model.

14. The method according to claim 13, wherein: The pruned visual large model includes M cascaded pruned image denoising networks, N>M>1, and M is an integer; The performing knowledge distillation on the pruned visual big model according to the trained visual big model comprises: Using the trained large visual model to process sample scene requirement text features and sample scene noise image features related to the specified scene, obtain the Nth level intermediate scene image fusion features and the Nth level intermediate scene text fusion features output by the Nth image denoising network; Processing the sample scene requirement text features and the sample scene noise image features using the pruned visual large model to obtain the Mth level intermediate pruned image fusion features and the Mth level intermediate pruned text fusion features output by the Mth pruned image denoising network; Determining a knowledge distillation loss based on the N-th level intermediate scene image fusion feature and the N-th level intermediate scene text fusion feature, and the M-th level intermediate pruning image fusion feature and the M-th level intermediate pruning text fusion feature; and The model parameters of the Mth pruned image denoising network are adjusted according to the knowledge distillation loss.

15. An image generation device based on a large language model, comprising: The first acquisition module is used to acquire the demand text; A first processing module is used to process the demand text using a large language model to obtain target demand text features, wherein the target demand text features include image word features and attribute prompt features, wherein the image word features represent image words that need to be displayed in the generated target image, and the attribute prompt features are related to image attributes of the image words; The second processing module is used to process the target demand text features using the visual large model to obtain the target image, wherein the target image includes an image word area displayed based on the image attributes, and the image word area represents the image words in the demand text.

16. The device according to claim 15, wherein: The visual large model includes N cascaded image denoising networks, N>1, and N is an integer; wherein the second processing module includes: The first image denoising processing submodule is used to process the target required text features and the preset noise features using the cascaded N image denoising networks based on the attention mechanism to obtain the target image.

17. The device according to claim 16, wherein: The first image noise reduction processing submodule includes: a first processing unit, configured to process the n-1th level intermediate text fusion feature and the n-1th level intermediate image fusion feature using the nth image denoising network based on an attention mechanism to obtain the nth level intermediate text fusion feature and the nth level intermediate image fusion feature, wherein N≥n>1, and n is an integer; and The second processing unit is used to determine the target image based on the Nth level intermediate image fusion feature when N=n.

18. The device according to claim 17, wherein: The first processing unit comprises: A first spatial domain modulation subunit is used to perform spatial domain modulation processing on the n-1th level intermediate text fusion feature and the noise feature based on preset noise information to obtain an nth spatial domain text feature and an nth spatial domain image feature corresponding to the preset spatial domain; a feature fusion subunit, configured to fuse the nth spatial domain text feature and the nth spatial domain image feature based on an attention mechanism to obtain an nth spatial domain text fusion feature and an nth spatial domain image fusion feature; and The second spatial domain modulation subunit is used to perform spatial domain modulation on the nth spatial domain text fusion feature and the nth spatial domain image fusion feature based on preset noise information to obtain the nth level intermediate text fusion feature and the nth level intermediate image fusion feature output by the nth image denoising network.

19. The device according to claim 17 or 18, wherein: The first-level intermediate text fusion features and the first-level intermediate image fusion features are determined by the first image denoising network processing the target required text features and the noise features.

20. The device according to claim 15, wherein: Also includes: An updating module, configured to update the target requirement text in response to the text editing request, and obtain an updated target requirement text, wherein the target requirement text is determined based on the target requirement text feature; as well as The third processing module is used to process the updated target requirement text features using the visual large model to obtain an updated target image, wherein the updated target requirement text features are determined based on the updated target requirement text.

21. The device according to claim 15, wherein: The image attributes include at least one of the following: Style attributes, texture attributes, font attributes, preset style attributes.

22. A training device for a large visual model, comprising: A second acquisition module is used to acquire sample target demand text features, sample noise images and label images, wherein the sample target demand text features are determined by processing the sample demand text using a large language model, and the sample target demand text features include sample image word features and sample attribute prompt features, wherein the sample image word features represent label image words displayed in the label image, and the attribute prompt features are related to label image attributes of the label image words, and the sample noise image is determined by performing noise processing on the label image; a fourth processing module, configured to process the sample target requirement text features and the sample noise image features using the initial visual large model to obtain a sample target image, wherein the sample target image includes a sample image word region associated with the label image word, and the sample noise image features are determined based on the sample noise image; as well as The training module is used to train the initial visual large model based on the sample target image and the label image to obtain a trained visual large model.

23. The device according to claim 22, wherein: The initial visual large model includes N cascaded initial image denoising networks, N>1, and N is an integer; Wherein, the fourth processing module includes: The second image denoising processing submodule is used to process the sample target required text features and the sample noise features based on the attention mechanism by utilizing the cascaded N initial image denoising networks to obtain the sample target image.

24. The device according to claim 22 or 23, wherein: The initial visual macro model is determined based on the following operations: Obtain a pre-trained visual model, wherein the pre-trained visual model includes K pre-trained image denoising networks, where K is an integer greater than 1; Copying the network structure of at least one of the pre-trained image denoising networks in the pre-trained visual model to obtain an extended image denoising network; and The pre-trained visual model is updated based on at least one of the extended image denoising networks to obtain an initial visual large model with N initial image denoising networks, N>K.

25. The device according to claim 24, wherein: The i-th initial image denoising network of the initial visual large model is the j-th pre-trained image denoising network of the pre-trained visual model, and the i+1-th initial image denoising network of the initial visual large model is the extended image denoising network corresponding to the j-th pre-trained image denoising network, where i and j are integers.

26. The device according to claim 24, wherein: The initial visual large model includes a cascaded i-th initial image denoising module and an i+1-th initial image denoising module; Among them, the multiple initial image denoising networks in the i-th initial image denoising module are multiple pre-trained image denoising networks of the pre-trained visual model, and the multiple initial image denoising networks in the i+1-th initial image denoising module are multiple extended image denoising networks; the multiple extended image denoising networks are determined by replicating the network structure of the multiple pre-trained image denoising networks.

27. The device according to claim 22, wherein: The visual large model includes N cascaded image denoising networks, N>1, and N is an integer; The device also includes: A pruning module, used for pruning the trained visual large model to obtain a pruned visual large model; and The knowledge distillation module is used to perform knowledge distillation on the pruned visual big model according to the trained visual big model to obtain an updated visual big model.

28. The device according to claim 27, wherein The pruned visual large model includes M cascaded pruned image denoising networks, N>M>1, and M is an integer; Wherein, the knowledge distillation module includes: A first processing submodule is used to process the sample scene requirement text features and the sample scene noise image features related to the specified scene using the trained visual large model to obtain the Nth level intermediate scene image fusion features and the Nth level intermediate scene text fusion features output by the Nth image denoising network; The second processing submodule is used to process the sample scene requirement text features and the sample scene noise image features using the pruned visual large model to obtain the Mth level intermediate pruned image fusion features and the Mth level intermediate pruned text fusion features output by the Mth pruned image denoising network; a loss determination submodule, configured to determine the knowledge distillation loss based on the N-th level intermediate scene image fusion feature and the N-th level intermediate scene text fusion feature, and the M-th level intermediate pruned image fusion feature and the M-th level intermediate pruned text fusion feature; and A training submodule is used to adjust the model parameters of the Mth pruned image denoising network according to the knowledge distillation loss.

29. An artificial intelligence agent, comprising: An input module, used for receiving input information; A processing module, configured to determine a target task based on the input information received by the input module, determine a large language model or a large visual model or an initial large visual model based on the target task, and obtain output information by calling the large language model and / or the large visual model to execute the method of any one of claims 1 to 7, or by calling the initial large visual model to execute the method of any one of claims 8 to 14; An output module is used to output the output information obtained by the processing module.

30. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

31. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-14.

32. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images