Image generation method based on large language model, training method and device of visual large model, and intelligent agent

Abstract

The disclosure provides a large language model-based image generation method, a visual large model training method, device and agent, and relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large model and the like. The large language model-based image generation method comprises: obtaining a demand text; processing the demand text by using a large language model to obtain target demand text features, the target demand text features comprising image word features and attribute prompt features, the image word features representing image words to be displayed in a generated target image, and the attribute prompt features being related to image attributes of the image words; and processing the target demand text features by using a visual large model to obtain a target image, the target image comprising an image word region displayed based on the image attributes, and the image word region representing the image words in the demand text.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of computer vision, deep learning, and large models, and can be applied to AIGC scenarios such as AI-based content generation. Background Technology

[0002] With the development of artificial intelligence technology, large models are increasingly being widely applied in various fields such as natural language processing, computer vision, and speech recognition. For example, large models can generate corresponding visual images based on natural language descriptions. Summary of the Invention

[0003] This disclosure provides an image generation method based on a large language model, a training method for a large visual model, an apparatus, a device, a medium, a program product, and an intelligent agent.

[0004] According to one aspect of this disclosure, an image generation method based on a large language model is provided, comprising: acquiring a requirement text; processing the requirement text using a large language model to obtain target requirement text features, the target requirement text features including image character features and attribute hint features, the image character features representing the image characters to be displayed in the generated target image, and the attribute hint features being related to the image attributes of the image characters; processing the target requirement text features using a visual large model to obtain a target image, the target image including an image character region displayed based on image attributes, the image character region representing the image characters in the requirement text.

[0005] According to another aspect of this disclosure, a method for training a large visual model is provided, comprising: acquiring sample target demand text features, sample noise images, and label images, wherein the sample target demand text features are determined by processing the sample demand text using a large language model, the sample target demand text features include sample image character features and sample attribute hint features, the sample image character features represent the label image characters displayed in the label image, the attribute hint features are related to the label image attributes of the label image characters, and the sample noise images are determined by adding noise to the label images; processing the sample target demand text features and sample noise image features using an initial large visual model to obtain a sample target image, the sample target image including sample image character regions related to the label image characters, and the sample noise image features being determined based on the sample noise images; and training an initial large visual model based on the sample target image and the label images to obtain a trained large visual model.

[0006] According to another aspect of this disclosure, an image generation apparatus based on a large language model is provided, comprising: a first acquisition module for acquiring requirement text; a first processing module for processing the requirement text using a large language model to obtain target requirement text features, the target requirement text features including image character features and attribute hint features, the image character features representing image characters to be displayed in the generated target image, and the attribute hint features being related to the image attributes of the image characters; and a second processing module for processing the target requirement text features using a visual large model to obtain the target image, the target image including an image character region displayed based on the image attributes, the image character region representing image characters in the requirement text.

[0007] According to another aspect of this disclosure, a training apparatus for a large visual model is provided, comprising: a second acquisition module for acquiring sample target demand text features, sample noise images, and label images, wherein the sample target demand text features are determined by processing the sample demand text using a large language model, the sample target demand text features include sample image character features and sample attribute hint features, the sample image character features characterizing the label image characters displayed in the label image, the attribute hint features being related to the label image attributes of the label image characters, and the sample noise image being determined by adding noise to the label image; a fourth processing module for processing the sample target demand text features and sample noise image features using an initial large visual model to obtain a sample target image, the sample target image including sample image character regions related to the label image characters, and the sample noise image features being determined based on the sample noise image; and a training module for training the initial large visual model based on the sample target image and the label image to obtain a trained large visual model.

[0008] According to another aspect of this disclosure, an artificial intelligence agent is provided, comprising: an input module for receiving input information; a processing module for determining a target task based on the input information received by the input module, determining a large language model or a large visual model or an initial large visual model based on the target task, executing the above-described method by calling the large language model and / or the large visual model, or executing the above-described method by calling the initial large visual model, to obtain output information; and an output module for outputting the output information obtained by the processing module.

[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0011] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0014] Figure 1 The illustration schematically shows an exemplary system architecture that can be applied to the image generation method and apparatus based on the large language model and the training method and apparatus for the large visual model according to embodiments of the present disclosure;

[0015] Figure 2 A flowchart illustrating an image generation method based on a large language model according to an embodiment of the present disclosure is shown schematically.

[0016] Figure 3A An example and a comparative example of a target image according to an embodiment of the present disclosure are illustrated;

[0017] Figure 3B Examples and comparative examples of icon images according to another embodiment of this disclosure are shown exemplarily;

[0018] Figure 4 A schematic diagram of a large visual model according to an embodiment of the present disclosure is shown.

[0019] Figure 5 A schematic diagram of a large visual model according to another embodiment of the present disclosure is shown;

[0020] Figure 6 A flowchart illustrating a method for training a large visual model according to an embodiment of the present disclosure is shown.

[0021] Figure 7A An illustrative illustration shows a label image according to an embodiment of the present disclosure;

[0022] Figure 7B An illustrative illustration shows a label image according to another embodiment of this disclosure;

[0023] Figure 8A An initial visual large model according to one embodiment of the present disclosure is illustrated by way of example;

[0024] Figure 8B An initial visual large model according to another embodiment of the present disclosure is illustrated by way of example;

[0025] Figure 9 The exemplary illustrations show the output effects of skipping layers 1, 2, 10, and 20 in the large visual model according to embodiments of the present disclosure;

[0026] Figure 10 The diagram illustrates a trained large visual model and a pruned large visual model according to embodiments of the present disclosure.

[0027] Figure 11 A block diagram of an image generation apparatus based on a large language model according to an embodiment of the present disclosure is shown schematically.

[0028] Figure 12 A block diagram of a training apparatus for a large visual model according to an embodiment of the present disclosure is shown schematically.

[0029] Figure 13 A schematic diagram illustrating the structure of an intelligent agent of artificial intelligence according to embodiments of the present disclosure; and

[0030] Figure 14 The diagram illustrates a block diagram of an electronic device suitable for implementing an image generation method based on a large language model and a training method for a large visual model, according to embodiments of the present disclosure. Detailed Implementation

[0031] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0032] In the technical solution disclosed herein, the acquisition, storage, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0033] With the development of artificial intelligence technology, large models can be applied to the field of computer vision. For example, large models can generate corresponding visual images based on natural language descriptions. However, the inventors found that large models perform poorly in generating text content that needs to be displayed in the generated images, making it difficult to generate suitable and semantically correct text content images.

[0034] To address this, this disclosure provides an image generation method based on a large language model, a training method for a large visual model, an apparatus, a device, a medium, a program product, and an intelligent agent. The image generation method based on a large language model includes: acquiring a requirement text; processing the requirement text using a large language model to obtain target requirement text features, the target requirement text features including image character features and attribute hint features, the image character features representing the image characters to be displayed in the generated target image, and the attribute hint features being related to the image attributes of the image characters; processing the target requirement text features using a large visual model to obtain a target image, the target image including an image character region displayed based on image attributes, the image character region representing the image characters in the requirement text.

[0035] According to embodiments of this disclosure, by processing the requirement text using a large language model, more comprehensive and accurate target requirement text features can be obtained to better characterize the image words to be displayed in the target image and the image attributes related to those image words. Attribute cue features can more accurately describe the image words to be displayed in the target image, helping the large visual model improve its semantic understanding capabilities. Based on this, the target requirement text features can be used to more accurately control the large visual model to understand the image words to be displayed in the target image. Furthermore, attribute cue features can be used to control the large visual model to generate the target image based on the image attributes of the image words, thereby improving the accuracy of the image words to be displayed in the target image, avoiding illusions when the large visual model processes the requirement text, and ensuring that the display effect of the image word region matches the image generation requirements of the requirement text, thus improving the overall imaging effect and accuracy of the target image.

[0036] According to embodiments of this disclosure, a large model can be a model with a large number of parameters, typically in the tens or hundreds of millions, possibly reaching the billions or tens of billions. Regarding the network structure of a large model, for example, a network structure such as UFO (Unified Feature Optimization) can be adopted.

[0037] Figure 1 The illustration schematically shows an exemplary system architecture that can be applied to image generation methods and apparatus based on large language models and training methods and apparatus for large visual models according to embodiments of the present disclosure.

[0038] It is important to note that Figure 1The examples shown are merely examples of system architectures applicable to embodiments of this disclosure, intended to help those skilled in the art understand the technical content of this disclosure. However, they do not imply that embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For instance, in another embodiment, an exemplary system architecture applicable to the image generation method and apparatus based on a large language model, and the training method and apparatus for a large visual model, may include a terminal device. However, the terminal device can implement the image generation method and apparatus based on a large language model, and the training method and apparatus for a large visual model provided in the embodiments of this disclosure without interacting with a server.

[0039] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0040] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0041] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0042] Server 105 can be a server that provides various services, such as a backend management server that supports the content browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0043] A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system. It solves the shortcomings of traditional physical hosts and VPS services ("Virtual Private Server", or simply "VPS"), such as high management difficulty and weak business scalability. A server can also be a server for a distributed system or a server that incorporates blockchain technology.

[0044] It should be noted that the image generation method and visual large model training method based on a large language model provided in this disclosure embodiment can generally be executed by server 105. Correspondingly, the image generation device and visual large model training device based on a large language model provided in this disclosure embodiment can generally be located in server 105. The image generation method and visual large model training method based on a large language model provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the image generation device and visual large model training device based on a large language model provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0045] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0046] Figure 2 A flowchart illustrating an image generation method based on a large language model according to an embodiment of the present disclosure is shown schematically.

[0047] like Figure 2 As shown, the method 200 includes operations S210 to S230.

[0048] In operation S210, obtain the required text.

[0049] In operation S220, the large language model is used to process the requirement text to obtain the target requirement text features. The target requirement text features include image word features and attribute hint features. Image word features represent the image words that need to be displayed in the generated target image, and attribute hint features are related to the image attributes of the image words.

[0050] In operation S230, the visual large model is used to process the target requirement text features to obtain the target image. The target image includes image word regions displayed based on image attributes, and the image word regions represent the image words in the requirement text.

[0051] According to embodiments of this disclosure, the requirement text may include natural language descriptions or keywords input by the user. The requirement text can be used to characterize the user's image generation intent, and it typically contains the content and features that the user hopes to see in the generated target image. For example, the requirement text can be in natural language such as Chinese or English, without specific limitations herein.

[0052] According to embodiments of this disclosure, the requirement text includes descriptive information related to the text content that needs to be displayed in the target image. More preferably, specific symbols (such as quotation marks, parentheses, separators, etc.) can be used to mark the text content in the requirement text.

[0053] As an example, text content can include language characters (such as Chinese and English characters), special symbols, punctuation marks, Greek characters, Latin characters, mathematical symbols, emoticons, serial numbers, units, and other textual information. For example, text content could be "ABCD", "tea", "No.1", etc.

[0054] As another example, the text content can also include logos, icons, trademarks, labels, special symbols, etc., that contain the aforementioned text information. For example, the text content could be "XXX icon" or "No Smoking sign," etc.

[0055] As another example, the text content can also include logos, icons, trademarks, labels, special symbols, etc., that can be described using the aforementioned text information. For example, the text content could be "Organization A's Logo" or "Product B's Trademark," etc.

[0056] For example, the requirement text could be: The image shows a coffee cup in a simple sketch style with the letters "ABCD" on the cup body. Another example is: The image shows a coffee cup in a simple sketch style with the letters "ABCD" on the cup body.

[0057] According to embodiments of this disclosure, a large language model can be used to process the requirement text to obtain target requirement text features. These features can include image-text features and attribute-hint features. Image-text features represent the image-text to be displayed in the generated target image, while attribute-hint features are related to the image attributes of the image-text. In one example, image-text can be understood as the text content described above.

[0058] As an example, the image attributes of image text may include style attributes, texture attributes, size attributes, position attributes, font attributes, etc., related to the text information mentioned above. As another example, the image attributes of image text may also include attribute information related to the text content of the preset styles such as logos, icons, trademarks, labels, and special symbols mentioned above.

[0059] According to embodiments of this disclosure, a Large Language Model (LLM) can be a natural language processing model with a large number of parameters and computational power. Large Language Models can be widely applied in fields such as natural language processing, text generation, and intelligent dialogue. A Large Language Model can be a model with a large number of parameters, typically on the order of tens or hundreds of millions, possibly reaching the order of billions or tens of billions. Regarding the network structure of a Large Language Model, for example, a Transformer network structure can be used.

[0060] According to one embodiment of this disclosure, a large language model can process the request text to obtain image character features and attribute hint features. The attribute hint features can more accurately describe the image characters to be displayed in the target image, thus helping the large visual model improve its semantic understanding capabilities. Based on this, the target request text features can be used to more accurately control the large visual model to understand the image characters to be displayed in the target image. Furthermore, the attribute hint features can be used to control the large visual model to generate the target image based on the image attributes of the image characters, thereby improving the accuracy of the image characters to be displayed in the target image, avoiding illusions when the large visual model processes the request text, and ensuring that the display effect of the image character region matches the image generation requirements of the request text, thus improving the overall imaging effect and accuracy of the target image.

[0061] According to embodiments of this disclosure, a target image can be obtained by processing the features of the target requirement text using a large visual model. The target image includes image-text regions displayed based on image attributes, where each image-text region represents an image-text element within the requirement text.

[0062] According to one embodiment of this disclosure, the image text displayed in the target image corresponds to the text content contained in the required text. For example, the image text region may include a region representing text information in the target image, which may include language characters (such as Chinese and English characters), special symbols, punctuation marks, Greek characters, Latin characters, mathematical symbols, emoticons, serial numbers, units, etc. As another example, the image text region may also include a region representing text content in the target image, which may include logos, icons, trademarks, labels, special marks, etc., containing the aforementioned text information or describable by the aforementioned text information.

[0063] According to embodiments of this disclosure, a large-scale visual model (Vision Transformer) can be a visual processing model with a large number of parameters and computational power. This large-scale visual model can extract useful features and information from visual data such as images and videos. It can be applied to technologies such as image recognition, object detection, semantic segmentation, and image generation. The large-scale visual model can be a model with a large number of parameters, typically in the tens or hundreds of millions, and possibly reaching the billions or tens of billions. Regarding the network structure of the large-scale visual model, for example, a Transformer, a Convolutional Neural Network (CNN), or a dual encoder network structure can be used.

[0064] According to embodiments of this disclosure, by processing the requirement text using a large language model, more comprehensive and accurate target requirement text features can be obtained to better characterize the image words to be displayed in the target image and the image attributes related to those image words. Attribute cue features can more accurately describe the image words to be displayed in the target image, helping the large visual model improve its semantic understanding capabilities. Based on this, the target requirement text features can be used to more accurately control the large visual model to understand the image words to be displayed in the target image. Furthermore, attribute cue features can be used to control the large visual model to generate the target image based on the image attributes of the image words, thereby improving the accuracy of the image words to be displayed in the target image, avoiding illusions when the large visual model processes the requirement text, and ensuring that the display effect of the image word region matches the image generation requirements of the requirement text, thus improving the overall imaging effect and accuracy of the target image.

[0065] Figure 3A An example and a comparative example of a target image according to one embodiment of the present disclosure are illustrated.

[0066] According to one embodiment of this disclosure, the requirement text can be: The image shows a coffee cup in a simple sketch style with the letters "ABCD" on the cup body. Alternatively, the requirement text can be: The image shows a coffee cup in a simple sketch style with the letters "ABCD" on the cup body.

[0067] exist Figure 3A In the above, target image 301 is a target image comparison example generated by a related method, and target image 302 is a target image example generated by the image generation method based on a large language model (hereinafter referred to as the method) provided in the embodiments of this disclosure.

[0068] Figure 3BAn example and a comparative illustration of an icon image according to another embodiment of the present disclosure are shown.

[0069] According to another embodiment of this disclosure, the requirement text can be: The image shows a straw cup in a simple sketch style, with the "XXX Product Logo" displayed at the center of the cup body. Alternatively, the requirement text can be: The image shows a straw cup in a simple sketch style, with the "XXX Product Logo" displayed at the center of the cup body.

[0070] exist Figure 3B In the above, target image 303 is a comparison of target images generated by a related method, and target image 304 is an example of a target image generated by this method.

[0071] like Figure 3A and Figure 3B As shown, compared with the target image generated by related methods, the target image generated by this method has a better effect in generating the text content to be displayed in the target image. This method can generate suitable and semantically correct text content images relatively well.

[0072] According to embodiments of this disclosure, the visual large model includes N cascaded image denoising networks, where N > 1 and N is an integer; wherein, processing the target demand text features using the visual large model to obtain the target image includes: processing the target demand text features and preset noise features using the cascaded N image denoising networks based on an attention mechanism to obtain the target image.

[0073] According to one embodiment of this disclosure, the large visual model can be selected as a diffusion model. The diffusion model can be used for image generation, and its inspiration comes from thermodynamics: a distribution can be transformed into another distribution by continuously adding noise. In the context of image generation tasks, this means that images from the training set can be transformed into images conforming to a standard normal distribution by continuously adding noise.

[0074] For example, the diffusion model consists of a forward process (or diffusion process) and a backward process (or inverse diffusion process), corresponding to encoding and decoding, respectively. In the forward process, noise is continuously added to the input image. After T noise addition operations, the input image becomes a pure noise image conforming to a standard normal distribution. In the backward process, we aim to train a denoising network that can learn to perform T denoising operations to restore the pure noise image back to the input image. Therefore, the learning objective of the denoising network is to ensure that the T denoising operations exactly cancel out the corresponding T noise addition operations. After training, we only need to randomly sample a noise from the standard normal distribution and then use the denoising network in the backward process to restore the noise into an image to generate a picture.

[0075] According to embodiments of this disclosure, the large visual model can be a diffusion model, and the image denoising network can be used as the reverse process of the diffusion model. An attention mechanism can be used to process the target text features and preset noise features using N cascaded image denoising networks to obtain the target image. As an example, the preset noise features can be a Gaussian noise matrix.

[0076] According to embodiments of this disclosure, unlike the encoder and decoder of traditional diffusion models, embodiments of this disclosure employ an attention-based DIT (Dual-Stream Diffusion Transformer Architecture) to process the target text features and noise features. In the DIT dual-stream architecture, two independent streams process image data and text data respectively, and then fuse them at a high level. Since the data of different modalities are processed independently, the feature information of each modality can be better preserved. Through independent processing and late-stage fusion, the large visual model can more effectively utilize information from different modalities, and better understand and utilize the complementary information between different modalities, thereby further improving the semantic understanding capability of the large visual model.

[0077] According to embodiments of this disclosure, by utilizing an attention mechanism, the image denoising network can better capture the target text features and preset noise features globally and locally, providing a more comprehensive feature representation. Compared to single-layer image denoising networks, multi-layer cascaded network structures can effectively enhance the model's feature representation capabilities, thereby helping to further improve the semantic understanding capabilities of the large visual model. This enhances the accuracy of the image text to be displayed in the target image, avoids visual illusions in the large visual model, and ensures that the display effect of the image text region matches the image generation requirements of the target text, thereby improving the overall imaging effect and accuracy of the target image.

[0078] According to embodiments of this disclosure, an attention mechanism can be used to process the target text features and preset noise features using cascaded N image denoising networks to obtain the target image. As an example only, Latent Flow can be used to process noise features, and Text Flow can be used to process the target text features.

[0079] According to embodiments of this disclosure, based on an attention mechanism, the target image is obtained by processing the target text features and preset noise features using N cascaded image denoising networks in a large visual model. This process includes: processing the (n-1)th level intermediate text fusion features and the (n-1)th level intermediate image fusion features using the nth image denoising network based on an attention mechanism to obtain the nth level intermediate text fusion features and the nth level intermediate image fusion features, where N ≥ n > 1 and n is an integer; and determining the target image based on the Nth level intermediate image fusion features when N = n.

[0080] Figure 4 A schematic diagram of a visual large model according to an embodiment of the present disclosure is shown.

[0081] According to one embodiment of this disclosure, such as Figure 4 As shown, the visual large model 400 may include two cascaded image denoising networks, namely a first image denoising network 410 and a second image denoising network 420.

[0082] According to one embodiment of this disclosure, such as Figure 4 As shown, based on the attention mechanism, two cascaded image denoising networks in the Visual Large Model 400 process the target text features and preset noise features to obtain the target image, which may include:

[0083] For example, when n=2, based on the attention mechanism, the second image denoising network 420 is used to process the first-level intermediate text fusion feature 403 and the first-level intermediate image fusion feature 404 to obtain the second-level intermediate text fusion feature 405 and the second-level intermediate image fusion feature 406.

[0084] For example, when n=2, the target image 407 is determined based on the second-level intermediate image fusion feature 406.

[0085] According to embodiments of this disclosure, based on an attention mechanism, processing the (n-1)th level intermediate text fusion features and the (n-1)th level intermediate image fusion features using the nth image denoising network to obtain the nth level intermediate text fusion features and the nth level intermediate image fusion features includes: performing spatial domain modulation processing on the (n-1)th level intermediate text fusion features and noise features based on preset noise information to obtain the nth spatial domain text features and the nth spatial domain image features corresponding to the preset spatial domain; fusing the nth spatial domain text features and the nth spatial domain image features based on an attention mechanism to obtain the nth spatial domain text fusion features and the nth spatial domain image fusion features; and performing spatial domain modulation on the nth spatial domain text fusion features and the nth spatial domain image fusion features based on preset noise information to obtain the nth level intermediate text fusion features and the nth level intermediate image fusion features output by the nth image denoising network.

[0086] According to embodiments of this disclosure, in a two-stream DiT architecture, spatial modulation processing can be understood as modulating features in a spatial dimension.

[0087] According to one embodiment of this disclosure, a large visual model may include three cascaded image denoising networks, namely a first image denoising network, a second image denoising network, and a third image denoising network.

[0088] For example, when n=2, the first-level intermediate text fusion features and noise features can be spatially modulated based on preset noise information to obtain second-level spatial domain text features and second-level spatial domain image features corresponding to the preset spatial domain. The second-level spatial domain text features and second-level spatial domain image features can be fused based on an attention mechanism to obtain second-level spatial domain text fusion features and second-level spatial domain image fusion features. The second-level spatial domain text fusion features and second-level spatial domain image fusion features can be spatially modulated based on preset noise information to obtain second-level intermediate text fusion features and second-level intermediate image fusion features output by the second image denoising network.

[0089] For example, when n=3, the second-level intermediate text fusion features and noise features can be spatially modulated based on preset noise information to obtain third-level spatial domain text features and third-level spatial domain image features corresponding to the preset spatial domain. The third-level spatial domain text features and third-level spatial domain image features can be fused based on an attention mechanism to obtain third-level spatial domain text fusion features and third-level spatial domain image fusion features. The third-level spatial domain text fusion features and third-level spatial domain image fusion features can be spatially modulated based on preset noise information to obtain the third-level intermediate text fusion features and third-level intermediate image fusion features output by the third image denoising network.

[0090] According to one embodiment of this disclosure, preset noise information can correspond to a preset noise level. Exemplarily, the noise level to be added to the (n-1)th level intermediate text fusion feature and noise feature can be determined based on the noise level. Optionally, the number of iterations (e.g., d) for processing the (n-1)th level intermediate text fusion feature and noise feature in the cascaded N image denoising networks in the large visual model can be determined based on the noise level (e.g., noise level t).

[0091] In one example, the preset noise information may include 10 preset noise information corresponding to noise level t=10, and the number of loops for processing the (n-1)th level intermediate text fusion feature and noise feature can be, for example, 10. In the first processing, the noise level t can be set to 10, indicating that the noise level added in the first processing is relatively high. As the number of loops increases, the noise level decreases accordingly. In the 10th processing, the noise level t can be set to 1, indicating that the noise level added in the last processing is relatively low.

[0092] It should be understood that the first iteration of the process can be understood as using N image denoising networks of the large visual model to process the target text features, preset noise features, and preset noise information to obtain the first round of image fusion features. The second iteration of the process can be understood as using N image denoising networks of the large visual model to process the first round of image fusion features, text fusion features, and preset noise information to obtain the second round of image fusion features and text fusion features.

[0093] According to embodiments of this disclosure, the first-level intermediate text fusion features and the first-level intermediate image fusion features are determined by the first image denoising network processing the target demand text features and the noise features.

[0094] According to embodiments of this disclosure, a first image denoising subnetwork can be used to process target demand text features and noise features to determine first-level intermediate text fusion features and first-level intermediate image fusion features.

[0095] According to one embodiment of this disclosure, spatial domain modulation processing can be performed on the target text features and noise features based on preset noise information to obtain first spatial domain text features and first spatial domain image features corresponding to the preset spatial domain. The first spatial domain text features and first spatial domain image features can be fused based on an attention mechanism to obtain first spatial domain text fusion features and first spatial domain image fusion features. Spatial domain modulation can be performed on the first spatial domain text fusion features and first spatial domain image fusion features based on preset noise information to obtain first-level intermediate text fusion features and first-level intermediate image fusion features output by the first image denoising network.

[0096] According to one embodiment of this disclosure, preset noise information can correspond to a preset noise level. Exemplarily, the degree of noise to be added to the target text features and the preset noise features can be determined based on the noise level. Optionally, the number of iterations (e.g., d) for processing the target text features and the preset noise features in the cascaded N image denoising networks in the large visual model can be determined based on the noise level (e.g., noise level t).

[0097] In one example, the preset noise information may include 10 preset noise information corresponding to noise level t=10. The number of loops for processing the target text features and the preset noise features can be, for example, 10. In the first processing iteration, the noise level t can be set to 10, indicating that the noise level added in the first processing is relatively high. As the number of loops increases, the noise level decreases accordingly. In the 10th processing iteration, the noise level t can be set to 1, indicating that the noise level added in the last processing iteration is relatively low.

[0098] Figure 5 A schematic diagram of a large visual model according to another embodiment of the present disclosure is shown.

[0099] According to one embodiment of this disclosure, such as Figure 5 As shown, the visual large model 500 may include two cascaded image denoising networks, namely the first image denoising network N510 and the second image denoising network N520.

[0100] According to one embodiment of this disclosure, such as Figure 5 As shown, based on the attention mechanism, processing the target text features and preset noise features using two cascaded image denoising networks in the Visual Large Model 500 can include the following operations:

[0101] For example, spatial domain modulation processing can be performed on the target requirement text feature 502 based on preset noise information 501 to process the target requirement text feature 502 from the text spatial domain to the preset spatial domain, thereby obtaining the first spatial domain text feature 504. For example, spatial domain modulation processing can be performed on the noise feature 503 based on preset noise information 501 to process the noise feature 503 from the image spatial domain to the preset spatial domain, thereby obtaining the first spatial domain image feature 505.

[0102] For example, text features 504 and image features 505 in the first spatial domain can be fused based on an attention mechanism to obtain text fusion features 506 and image fusion features 507 in the first spatial domain. This enhances the image denoising network's ability to capture and understand features from different modalities, thereby improving the performance and effectiveness of large-scale visual models.

[0103] For example, the first spatial domain text fusion feature 506 and the first spatial domain image fusion feature 507 can be spatially modulated based on the preset noise information 501 to obtain the first-level intermediate text fusion feature 508 and the first-level intermediate image fusion feature 509 output by the first image denoising network N510.

[0104] For example, the first spatial domain text fusion feature 506 can be spatially modulated based on preset noise information 501 to process the first spatial domain text fusion feature 506 from the preset spatial domain to the text spatial domain, thereby obtaining the first-level intermediate text fusion feature 508 output by the first image denoising network N510. For example, the first spatial domain image fusion feature 507 can be spatially modulated based on preset noise information 501 to process the first spatial domain image fusion feature 507 from the preset spatial domain to the image spatial domain, thereby obtaining the first-level intermediate image fusion feature 509 output by the first image denoising network N510.

[0105] For example, the first-level intermediate text fusion feature 508 can be spatially modulated based on preset noise information 501 to process the first-level intermediate text fusion feature 508 from the text spatial domain to the preset spatial domain, thereby obtaining the second spatial domain text feature 510. For example, the first-level intermediate text fusion feature 508 can be spatially modulated based on preset noise information 501 to process the first-level intermediate text fusion feature 508 from the image spatial domain to the preset spatial domain, thereby obtaining the second spatial domain image feature 511.

[0106] For example, second-space domain text features 510 and second-space domain image features 511 can be fused based on an attention mechanism to obtain second-space domain text fusion features 512 and second-space domain image fusion features 513. Based on the attention mechanism, second-space domain text features 510 and second-space domain image features 511 can be fused to obtain second-space domain text fusion features 512 and second-space domain image fusion features 513, thereby enhancing the image denoising network's ability to capture and understand features of different modalities, thus improving the performance and effectiveness of large-scale visual models.

[0107] For example, the second spatial domain text fusion feature 512 can be spatially modulated based on preset noise information 501 to transform the second spatial domain text fusion feature 512 from the preset spatial domain to the text spatial domain, thereby obtaining the second-level intermediate text fusion feature 514 output by the second image denoising network N520. For example, the second spatial domain image fusion feature 513 can be spatially modulated based on preset noise information 501 to transform the second spatial domain image fusion feature 513 from the preset spatial domain to the image spatial domain, thereby obtaining the second-level intermediate image fusion feature 515 output by the second image denoising network N520.

[0108] For example, the second-level intermediate text fusion feature 514 and the second-level intermediate image fusion feature 515 output by the second image denoising network N520 can be input into the first image denoising network N510 for further processing until iteration d times, at which point the final visual large model 500 outputs the target image 516. The number of iterations d is determined based on the noise level t represented by the preset noise information 501.

[0109] According to embodiments of this disclosure, the image generation method based on a large language model further includes: responding to a text editing request, updating the target requirement text to obtain an updated target requirement text, wherein the target requirement text is determined based on target requirement text features; and processing the updated target requirement text features using a large visual model to obtain an updated target image, wherein the updated target requirement text features are determined based on the updated target requirement text.

[0110] According to one embodiment of this disclosure, the generated target image can be fed back to the user. The user can then update the request text to adjust the target image and trigger a text editing request.

[0111] According to embodiments of this disclosure, in response to a text editing request, a large language model can be used to process the updated requirement text to update the target requirement text, resulting in an updated target requirement text. The target requirement text is determined based on target requirement text features. For example, the target requirement text can be a prompt obtained by the large language model based on the target requirement text features, which is more accurate and contains descriptive information related to the text content that needs to be displayed in the target image. Updated target requirement text features can be determined based on the updated target requirement text. The updated target requirement text features can be processed using a large visual model to obtain an updated target image.

[0112] According to embodiments of this disclosure, by updating the target requirement text, the target requirement text can be further adjusted based on user feedback to improve the accuracy of the target requirement text features. This allows for more accurate control of the visual large model to understand the image text that needs to be displayed in the target image. Furthermore, the visual large model can be controlled based on attribute hint features to generate the target image based on image attributes, thereby improving the accuracy of the image text to be displayed in the target image, avoiding illusions when the visual large model processes the requirement text, and ensuring that the display effect of the image text area matches the image generation requirements of the requirement text, thus improving the overall imaging effect and imaging accuracy of the target image.

[0113] According to embodiments of this disclosure, image attributes include at least one of the following: style attributes, texture attributes, font attributes, and preset style attributes.

[0114] As an example, font attributes can characterize the font size, font, style, color, and so on of text information. Preset style attributes can characterize the attribute information related to the text content of preset styles such as logos, icons, trademarks, labels, and special symbols.

[0115] Figure 6 A flowchart illustrating a method for training a large visual model according to an embodiment of the present disclosure is shown.

[0116] like Figure 6 As shown, the method 600 includes operations S610~S620.

[0117] In operation S610, sample target requirement text features, sample noise images, and label images are obtained. Sample target requirement text features are determined by processing the sample requirement text using a large language model. Sample target requirement text features include sample image word features and sample attribute hint features. Sample image word features represent the label image words displayed in the label image. Attribute hint features are related to the label image attributes of the label image words. Sample noise images are determined by adding noise to the label images.

[0118] In operation 620, the initial visual large model is used to process the sample target requirement text features and sample noise image features to obtain the sample target image. The sample target image includes the sample image character region related to the label image character. The sample noise image features are determined based on the sample noise image.

[0119] In operation 630, an initial large visual model is trained based on sample target images and label images to obtain the trained large visual model.

[0120] According to embodiments of this disclosure, a label image displays label image text, and a sample noise image can be obtained by adding noise to the label image. For example, the sample noise image can be a Gaussian noise image.

[0121] According to embodiments of this disclosure, sample target requirement text features can be obtained based on large language processing sample requirement text. The sample target requirement text features may include sample image word features and sample attribute hint features. Specifically, the sample image word features characterize the label image words displayed in the label image, and the attribute hint features are related to the label image attributes of the label image words.

[0122] As an example, the label image attributes of a label image text may include style attributes, texture attributes, size attributes, position attributes, font attributes, and so on, related to the text information described above. As another example, the label image attributes of a label image text may also include attribute information related to text content in preset styles such as LOGOs, icons, trademarks, labels, and special symbols described above.

[0123] According to embodiments of this disclosure, an initial large visual model can be trained based on sample target images and label images to obtain a trained large visual model.

[0124] According to one embodiment of this disclosure, a sample set can be constructed based on label images and label text. Exemplarily, the sample set may include a first sample set and a second sample set.

[0125] As an example, the first sample set can be constructed as follows:

[0126] a. Targeted collection and display of label images containing text information, which can be understood as the text in the label images mentioned above. For example, text information may include language characters (such as Chinese and English characters), special symbols, punctuation marks, Greek characters, Latin characters, mathematical symbols, emoticons, serial numbers, units, etc.;

[0127] b. Using algorithms such as optical character recognition to identify text information in label images;

[0128] c. Use a large language model to process the recognized text information and image descriptions of the label images to obtain the label text with embedded text information, and mark the text information in the label text with specific symbols (such as quotation marks).

[0129] Figure 7A An illustrative label image is shown according to one embodiment of the present disclosure.

[0130] According to one embodiment of this disclosure, exemplarily, the label text could be: In this cartoon-style sticker image, a light beige background gently envelops every detail. At the bottom of the image, a striking line of red capital letters “GO AWAY IM READING” is particularly eye-catching. This text not only contrasts sharply with the background but also conveys a tranquil and focused atmosphere, as if telling the outside world that the child is enjoying their own reading time and does not want to be disturbed.

[0131] As another example, the second sample set can be constructed as follows:

[0132] a. Targeted collection and display of label images containing text content, which can be understood as the aforementioned label image text. For example, the text content may include logos, icons, trademarks, labels, special marks, etc., containing the aforementioned text information or that can be described by the aforementioned text information;

[0133] b. Based on the preset list and recognition operators, the text content in the collected label images is recognized to obtain the text content description information;

[0134] c. For labeled images where no text content is identified, collect descriptive information using a retrieval system and manually annotate the descriptive text content.

[0135] d. Use a large language model to process the identified text content description information and the image description about the label image to obtain the label text with embedded text content description information, and mark the text content description information in the label text with specific symbols (such as quotation marks).

[0136] Figure 7B An illustrative image of a label according to another embodiment of this disclosure is shown.

[0137] According to one embodiment of this disclosure, exemplarily, the sample text could be: a white yogurt bottle labeled "XYZ," with a simple design and gold and red decorative patterns and text on the front. The main patterns are cherry and rose elements, and the text includes information such as flavor and net content. The bottle appears wider at the bottom and has a neck-shaped top, with a white plastic cap. The "XYZ" logo is located in the center of the image.

[0138] According to embodiments of this disclosure, the initial visual large model includes N cascaded initial image denoising networks, where N > 1 and N is an integer; wherein, processing the target demand text features and sample noise image features using the initial visual large model to obtain the sample target image includes: processing the sample target demand text features and sample noise features using the cascaded N initial image denoising networks based on an attention mechanism to obtain the sample target image.

[0139] According to embodiments of this disclosure, the initial large visual model can be a diffusion model, and the initial image denoising network can be used as the reverse process of the diffusion model. An attention mechanism can be used to process the sample target text features and sample noise features using N cascaded initial image denoising networks to obtain the sample target image. As an example, the sample noise features can be a Gaussian noise matrix.

[0140] According to embodiments of this disclosure, the inventors have discovered that increasing the number of parameters in a large visual model has a positive effect on the generation effect of the target image. However, the relevant large visual models are often trained from scratch after the number of parameters is increased, resulting in a long training time.

[0141] According to embodiments of this disclosure, the initial large visual model is determined based on the following operations: obtaining a pre-trained visual model, wherein the pre-trained visual model includes K pre-trained image denoising networks, where K is an integer greater than 1; performing network structure replication on at least one pre-trained image denoising network in the pre-trained visual model to obtain an extended image denoising network; and updating the pre-trained visual model based on at least one extended image denoising network to obtain an initial large visual model with N initial image denoising networks, where N > K.

[0142] According to embodiments of this disclosure, it is understood that both the pre-trained image denoising network and the extended image denoising network are pre-trained networks with a certain generalization ability. Therefore, in subsequent model training tasks, the model can converge faster, thereby accelerating the training process.

[0143] According to embodiments of this disclosure, by utilizing a pre-trained denoising network and an extended image denoising network, the number of parameters in the initial large visual model can be quickly and conveniently expanded, which helps to improve the performance of the large visual model in generating target images. By updating the pre-trained visual model based on the extended image denoising network, the number of parameters that need to be retrained can be reduced while maintaining the total number of model parameters, thereby effectively reducing the training time and the consumption of computing resources.

[0144] According to embodiments of this disclosure, the i-th initial image denoising network of the initial large visual model is the j-th pre-trained image denoising network of the pre-trained visual model, and the (i+1)-th initial image denoising network of the initial large visual model is the extended image denoising network corresponding to the j-th pre-trained image denoising network, where i and j are integers.

[0145] Figure 8A An initial visual large model according to one embodiment of the present disclosure is illustrated.

[0146] According to one embodiment of this disclosure, such as Figure 8AAs shown, the pre-trained visual model can include three pre-trained image denoising networks, namely layer1, layer2, and layer3. The network structures of these three pre-trained image denoising networks can be copied to obtain three extended image denoising networks, namely layer1', layer2', and layer3'. These can be stacked in the pattern of layer1, layer1', layer2, layer2', layer3, layer3' to obtain the initial large visual model 810.

[0147] exist Figure 8A For example, the third initial image denoising network of the initial large visual model 810 is the first pre-trained image denoising network layer1 of the pre-trained visual model, and the fourth initial image denoising network of the initial large visual model is the extended image denoising network layer1' corresponding to the first pre-trained image denoising network layer1.

[0148] According to embodiments of this disclosure, the initial large visual model includes a cascaded i-th initial image denoising module and an i+1-th initial image denoising module; wherein, the multiple initial image denoising networks in the i-th initial image denoising module are multiple target pre-trained image denoising networks of the pre-trained visual model, and the multiple initial image denoising networks in the i+1-th initial image denoising module are multiple associated extended image denoising networks; the multiple associated extended image denoising networks are determined by replicating the network structure of the multiple target pre-trained image denoising networks.

[0149] Figure 8B An initial visual large model according to another embodiment of this disclosure is illustrated.

[0150] According to one embodiment of this disclosure, such as Figure 8B As shown, the pre-trained visual model can include three pre-trained image denoising networks, namely layer1, layer2, and layer3. The network structures of these three target pre-trained image denoising networks can be copied to obtain three extended image denoising networks, namely layer1', layer2', and layer3'. These can be stacked in the pattern of layer1, layer2, layer3, layer1', layer2', and layer3' to obtain the initial large visual model 800.

[0151] exist Figure 8BIn the initial visual large model 8200, there are cascaded first initial image denoising module 821 and second initial image denoising module 822. layer1, layer2 and layer3 constitute the first initial image denoising module 821, and layer1', layer2' and layer3' constitute the second initial image denoising module 822.

[0152] According to embodiments of this disclosure, the inventors also discovered that large visual models with a large number of parameters perform poorly under limited computing resources, resulting in excessively long target image generation times. Furthermore, the inventors found that by pruning the parameters of the large visual model, it is possible to effectively improve image generation speed while minimizing the decrease in image generation quality.

[0153] According to one embodiment of this disclosure, the inventors observed the impact of each layer of the visual large model on the overall output effect by skipping the output effect of a certain layer in the visual large model when outputting the target image.

[0154] Figure 9 The illustrations exemplify the output effects of skipping layers 1, 2, 10, and 20 of a large visual model according to embodiments of this disclosure. For example, the prompt is: three medical personnel, female, 18 years old, with short black hair, smiling, standing demurely, against a hospital background. Figure 9 It can be observed that skipping layers 10 and 20 has little impact on the overall output image quality.

[0155] According to embodiments of this disclosure, the visual large model includes N cascaded image denoising networks, where N > 1 and N is an integer; the training method of the visual large model further includes: pruning the trained visual large model to obtain a pruned visual large model; and performing knowledge distillation on the pruned visual large model based on the trained visual large model to obtain an updated visual large model.

[0156] According to embodiments of this disclosure, pruning the trained large visual model can effectively reduce the number of parameters in the large visual model, thereby reducing energy consumption and improving image generation speed. By performing knowledge distillation on the pruned large visual model based on the trained large visual model, knowledge from the large visual model with a large number of parameters can be transferred to the pruned large visual model with a smaller number of parameters. This allows the pruned large visual model to maintain or approach the accuracy and performance of the original model while being lightweight, ensuring that the pruned large visual model can generate better target images.

[0157] According to embodiments of this disclosure, the pruned visual large model includes M cascaded pruned image denoising networks, where N > M > 1, and M is an integer. The knowledge distillation of the pruned visual large model based on the trained visual large model includes: processing sample scene requirement text features and sample scene noise image features related to a specified scene using the trained visual large model to obtain the Nth-level intermediate scene image fusion feature and the Nth-level intermediate scene text fusion feature output by the Nth image denoising network; processing the sample scene requirement text features and sample scene noise image features using the pruned visual large model to obtain the Mth-level intermediate pruned image fusion feature and the Mth-level intermediate pruned text fusion feature output by the Mth pruned image denoising network; determining the knowledge distillation loss based on the Nth-level intermediate scene image fusion feature and the Nth-level intermediate scene text fusion feature, as well as the Mth-level intermediate pruned image fusion feature and the Mth-level intermediate pruned text fusion feature; and adjusting the model parameters of the Mth pruned image denoising network according to the knowledge distillation loss.

[0158] According to embodiments of this disclosure, the sample scene requirement text features and sample scene noise image features are determined based on the sample scene requirement text and sample scene image related to a specified scene.

[0159] Figure 10 Exemplary examples are shown of a trained large visual model and a pruned large visual model according to embodiments of the present disclosure.

[0160] According to one embodiment of this disclosure, such as Figure 10 As shown, the trained large visual model 1010 can include 20 cascaded image denoising networks (block1~block20). The trained large visual model 1010 can be pruned to obtain the pruned large visual model 1020. Removing blocks10~20 from the trained large visual model 1010 yields the pruned large visual model 1020. The pruned large visual model 1020 can include 10 cascaded image denoising networks (block1~block10).

[0161] As an example, such as Figure 10 As shown, the trained large visual model 1010 can be used to process the sample scene requirement text features 1001 and sample scene noise image features 1002 related to a specified scene, resulting in the 20th level intermediate scene text fusion features 1003 and 20th level intermediate scene image fusion features 1004 output by the 20th image denoising network (block20). The sample scene requirement text features 1001 and sample scene noise image features 1002 are determined based on the sample scene requirement text and sample scene image related to the specified scene.

[0162] As an example, such as Figure 10As shown, the pruning visual large model 1020 can be used to process the sample scene requirement text features 1001 and sample scene noise image features 1002 to obtain the 10th level intermediate pruning text fusion features 1005 and 10th level intermediate pruning image fusion features 1006 output by the 10th pruning image denoising network (block10).

[0163] As an example, such as Figure 10 As shown, the knowledge distillation loss can be determined based on the 20th-level intermediate scene text fusion feature 1003 and the 20th-level intermediate scene image fusion feature 1004, as well as the 10th-level intermediate pruned text fusion feature 1005 and the 10th-level intermediate pruned image fusion feature 1006. For example, the knowledge distillation text loss 1007 can be determined based on the 20th-level intermediate scene text fusion feature 1003 and the 10th-level intermediate pruned text fusion feature 1005. The knowledge distillation image loss 1008 can be determined based on the 20th-level intermediate scene image fusion feature 1004 and the 10th-level intermediate pruned image fusion feature 1006. The knowledge distillation loss 1009 can be determined based on the knowledge distillation image loss and the knowledge distillation text loss.

[0164] As an example, such as Figure 10 As shown, the model parameters of the 10th pruned image denoising network (block10) can be adjusted based on the knowledge distillation loss 1009.

[0165] Figure 11 A block diagram of an image generation apparatus based on a large language model according to an embodiment of the present disclosure is shown schematically.

[0166] like Figure 11 As shown, the image generation device 1100 based on a large language model may include a first acquisition module 1110, a first processing module 1120, and a second processing module 1130.

[0167] The first acquisition module 1110 is used to acquire the required text.

[0168] The first processing module 1120 is used to process the requirement text using a large language model to obtain target requirement text features. The target requirement text features include image word features and attribute hint features. The image word features represent the image words that need to be displayed in the generated target image, and the attribute hint features are related to the image attributes of the image words.

[0169] The second processing module 1130 is used to process the target requirement text features using a large visual model to obtain the target image. The target image includes an image word region displayed based on the image attributes, and the image word region represents the image words in the requirement text.

[0170] According to embodiments of this disclosure, the large visual model includes N cascaded image denoising networks, where N > 1 and N is an integer; wherein the second processing module includes a first image denoising processing submodule.

[0171] The first image denoising processing submodule is used to process the target text features and preset noise features based on the attention mechanism using N cascaded image denoising networks to obtain the target image.

[0172] According to embodiments of this disclosure, the first image noise reduction processing submodule includes a first processing unit and a second processing unit.

[0173] The first processing unit is used to process the (n-1)th level intermediate text fusion features and the (n-1)th level intermediate image fusion features using the nth image denoising network based on the attention mechanism, so as to obtain the nth level intermediate text fusion features and the nth level intermediate image fusion features, where N≥n>1 and n is an integer. The first level intermediate text fusion features and the first level intermediate image fusion features are determined by the first image denoising network processing the target requirement text features and noise features.

[0174] The second processing unit is used to determine the target image based on the fusion features of the Nth level intermediate image when N=n.

[0175] According to embodiments of this disclosure, the first processing unit may include a first spatial domain modulation subunit, a feature fusion subunit, and a second spatial domain modulation subunit.

[0176] The first spatial domain modulation subunit is used to perform spatial domain modulation processing on the (n-1)th level intermediate text fusion features and noise features based on preset noise information to obtain the nth spatial domain text features and nth spatial domain image features corresponding to the preset spatial domain.

[0177] The feature fusion subunit is used to fuse text features and image features in the nth spatial domain based on an attention mechanism to obtain text fusion features and image fusion features in the nth spatial domain.

[0178] The second spatial domain modulation subunit is used to perform spatial domain modulation on the nth spatial domain text fusion feature and the nth spatial domain image fusion feature based on preset noise information, so as to obtain the nth level intermediate text fusion feature and the nth level intermediate image fusion feature output by the nth image denoising network.

[0179] According to embodiments of this disclosure, the first-level intermediate text fusion features and the first-level intermediate image fusion features are determined by the first image denoising network processing the target demand text features and the noise features.

[0180] According to embodiments of this disclosure, the image generation apparatus based on a large language model further includes an update module and a third processing module.

[0181] The update module is used to update the target requirement text in response to a text editing request, and obtain the updated target requirement text, wherein the target requirement text is determined based on the characteristics of the target requirement text.

[0182] The third processing module is used to process the updated target requirement text features using a large visual model to obtain an updated target image. The updated target requirement text features are determined based on the updated target requirement text.

[0183] According to embodiments of this disclosure, image attributes include at least one of the following: style attributes, texture attributes, font attributes, and preset style attributes.

[0184] Figure 12 A block diagram of a training apparatus for a large visual model according to an embodiment of the present disclosure is shown schematically.

[0185] like Figure 12 As shown, the training device 1200 for the large visual model may include a second acquisition module 1210, a fourth processing module 1220, and a training module 1230.

[0186] The second acquisition module 1210 is used to acquire sample target demand text features, sample noise images, and label images. The sample target demand text features are determined by processing the sample demand text using a large language model. The sample target demand text features include sample image character features and sample attribute hint features. The sample image character features represent the label image characters displayed in the label image. The attribute hint features are related to the label image attributes of the label image characters. The sample noise image is determined by adding noise to the label image.

[0187] The fourth processing module 1220 is used to process the sample target requirement text features and sample noise image features using the initial visual large model to obtain a sample target image. The sample target image includes a sample image character region related to the label image character. The sample noise image features are determined based on the sample noise image.

[0188] The training module 1230 is used to train the initial large visual model based on the sample target image and the label image to obtain the trained large visual model.

[0189] According to embodiments of this disclosure, the initial visual large model includes N cascaded initial image denoising networks, where N > 1 and N is an integer; wherein, the fourth processing module includes a second image denoising processing submodule.

[0190] The second image denoising processing submodule is used to process the sample target text features and sample noise features using a cascaded N initial image denoising networks based on an attention mechanism, in order to obtain the sample target image.

[0191] According to embodiments of this disclosure, the initial large visual model is determined based on the following operations:

[0192] Obtain a pre-trained visual model, which includes K pre-trained image denoising networks, where K is an integer greater than 1;

[0193] The network structure of at least one pre-trained image denoising network in the pre-trained visual model is copied to obtain an extended image denoising network; and

[0194] The pre-trained visual model is updated based on at least one extended image denoising network to obtain an initial large visual model with N initial image denoising networks, where N > K.

[0195] According to embodiments of this disclosure, the i-th initial image denoising network of the initial large visual model is the j-th pre-trained image denoising network of the pre-trained visual model, and the (i+1)-th initial image denoising network of the initial large visual model is the extended image denoising network corresponding to the j-th pre-trained image denoising network, where i and j are integers.

[0196] According to embodiments of this disclosure, the initial large visual model includes a cascaded i-th initial image denoising module and an i+1-th initial image denoising module; wherein, the multiple initial image denoising networks in the i-th initial image denoising module are multiple pre-trained image denoising networks of the pre-trained visual model, and the multiple initial image denoising networks in the i+1-th initial image denoising module are multiple extended image denoising networks; the multiple extended image denoising networks are determined by copying the network structure of the multiple pre-trained image denoising networks.

[0197] According to embodiments of this disclosure, the visual large model includes N cascaded image denoising networks, where N > 1 and N is an integer; the training device for the visual large model also includes a pruning module and a knowledge distillation module.

[0198] The pruning module is used to prune the trained large visual model to obtain a pruned large visual model.

[0199] The knowledge distillation module is used to perform knowledge distillation on the pruned visual model based on the trained visual model, so as to obtain an updated visual model.

[0200] According to embodiments of this disclosure, the pruning visual large model includes M cascaded pruning image denoising networks, where N > M > 1, and M is an integer; wherein, the knowledge distillation module includes a first processing submodule, a second processing submodule, a loss determination submodule, and a training submodule.

[0201] The first processing submodule is used to process the sample scene requirement text features and sample scene noise image features related to the specified scene using the trained large visual model, so as to obtain the Nth level intermediate scene image fusion features and Nth level intermediate scene text fusion features output by the Nth image denoising network.

[0202] The second processing submodule is used to process the text features of the sample scene requirements and the image features of the sample scene noise using the pruning visual large model, so as to obtain the Mth level intermediate pruning image fusion feature and the Mth level intermediate pruning text fusion feature output by the Mth pruning image denoising network.

[0203] The loss determination submodule is used to determine the knowledge distillation loss based on the Nth-level intermediate scene image fusion features and the Nth-level intermediate scene text fusion features, as well as the Mth-level intermediate pruned image fusion features and the Mth-level intermediate pruned text fusion features.

[0204] The training submodule is used to adjust the model parameters of the Mth pruned image denoising network based on the knowledge distillation loss.

[0205] Figure 13 A schematic block diagram of an artificial intelligence agent according to an embodiment of the present disclosure is shown.

[0206] In embodiments of this disclosure, such as Figure 13 As shown, the AI ​​agent 1300 may include an input module 1310, a processing module 1320, and an output module 1330.

[0207] Input module 1310 is used to receive input information.

[0208] The processing module 1320 is used to determine the target task based on the input information received by the input module, determine the large language model or the large visual model or the initial large visual model based on the target task, and obtain the output information by calling the large language model and / or the large visual model to execute the above-mentioned image generation method based on the large language model, or by calling the initial large visual model to execute the above-mentioned training method of the large visual model.

[0209] Output module 1330 is used to output the output information obtained by the processing module.

[0210] According to embodiments of this disclosure, the input module 1310 is responsible for receiving or sensing information such as queries, requests, instructions, signals, or data from the outside world (e.g., users or the external environment), and converting it into a format that the AI ​​agent 1300 can understand and process. The input module 1310 is the primary link for the AI ​​agent 1300 to interact with the outside world, enabling the AI ​​agent 1300 to efficiently and accurately obtain necessary "sensory" information from the outside world and respond to this information.

[0211] In the example, input module 1310 can input the requirement text described above or the sample target requirement text features, sample noise image and label image.

[0212] In the example, processing module 1320 is the core support for the AI ​​agent 1300's ability to handle complex tasks. Processing module 1320 can execute the image generation method based on a large language model and the training method of a large visual model described above.

[0213] In the example, the performance of processing module 1320 is closely related to the large model on which AI agent 1300 is based. To fully leverage the capabilities of the large model, the internal structure of processing module 1320 can be designed to be highly configurable and scalable to handle various types of tasks and requirements in real-world scenarios.

[0214] In the example, after the AI ​​agent 1300 acquires the requirement text, the processing module 1320 can use a large language model to process the requirement text and obtain the target requirement text features. The processing module 1320 can then use a large visual model to process the target requirement text features, obtain the target image, and pass the target image to the output module 1330.

[0215] In the example, after the AI ​​agent 1300 acquires the sample target requirement text features, sample noise image, and label image, the processing module 1320 can use the initial large visual model to process the sample target requirement text features and sample noise image features to obtain the sample target image. The processing module 1320 can train the initial large visual model based on the sample target image and label image to obtain the trained large visual model, and then pass the trained large visual model to the output module 1330.

[0216] Understandably, while large language models possess excellent language understanding and generation capabilities, like humans, their ability to solve tasks is limited without the aid of any tools. However, once the AI ​​agent 1300 is given the ability to invoke tools, it can perform tasks such as using a calculator to complete mathematical calculations, using Python to perform data analysis, and using a search engine to generate weather forecasts.

[0217] In the example, output module 1330 can output the target image or the trained large visual model described above.

[0218] The AI ​​agent 1300 according to the embodiments of this disclosure can simply and effectively improve the level of intelligence, as well as enhance flexibility and versatility.

[0219] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0220] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0221] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0222] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0223] Figure 14 A schematic block diagram of an example electronic device 1400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0224] like Figure 14 As shown, device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1402 or a computer program loaded from storage unit 1408 into random access memory (RAM) 1403. The RAM 1403 may also store various programs and data required for the operation of device 1400. The computing unit 1401, ROM 1402, and RAM 1403 are interconnected via bus 1404. Input / output (I / O) interface 1405 is also connected to bus 1404.

[0225] Multiple components in device 1400 are connected to I / O interface 1405, including: input unit 1406, such as a keyboard, mouse, etc.; output unit 1407, such as various types of displays, speakers, etc.; storage unit 1408, such as a disk, optical disk, etc.; and communication unit 1409, such as a network card, modem, wireless transceiver, etc. Communication unit 1409 allows device 1400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0226] The computing unit 1401 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1401 performs the various methods and processes described above, such as image generation methods based on large language models and training methods for large visual models. For example, in some embodiments, the image generation methods based on large language models and the training methods for large visual models can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1408. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1400 via ROM 1402 and / or communication unit 1409. When the computer program is loaded into RAM 1403 and executed by the computing unit 1401, one or more steps of the image generation methods based on large language models and the training methods for large visual models described above can be performed. Alternatively, in other embodiments, computing unit 1401 may be configured by any other suitable means (e.g., by means of firmware) to perform image generation methods based on large language models or training methods for large visual models.

[0227] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0228] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0229] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0230] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0231] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0232] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0233] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0234] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image generation method based on a large language model, comprising: Obtain the required text; The requirement text is processed using a large language model to obtain target requirement text features, which include image word features and attribute hint features. The image word features represent the image words that need to be displayed in the generated target image, and the attribute hint features are related to the image attributes of the image words. The target image is obtained by processing the features of the target requirement text using a large visual model. The target image includes image word regions displayed based on the image attributes, and the image word regions represent image words in the requirement text. The large visual model includes N cascaded image denoising networks, and the image denoising networks determine the target image based on the following operations: Based on preset noise information, spatial domain modulation processing is performed on the intermediate text fusion features and noise features of level n-1 to obtain the text features and image features of the nth spatial domain corresponding to the preset spatial domain, where N≥n>1 and N and n are integers; Based on the attention mechanism, the text features and image features of the nth spatial domain are fused to obtain the text fusion features and image fusion features of the nth spatial domain; Based on preset noise information, the nth spatial domain text fusion feature and the nth spatial domain image fusion feature are spatially modulated to obtain the nth level intermediate text fusion feature and the nth level intermediate image fusion feature output by the nth image denoising network; When N=n, the target image is determined based on the Nth level intermediate image fusion features.

2. The method according to claim 1, wherein, The first-level intermediate text fusion features and the first-level intermediate image fusion features are determined by the first image denoising network processing the target text features and the noise features.

3. The method according to claim 1, wherein, Also includes: In response to a text editing request, the target requirement text is updated to obtain an updated target requirement text, wherein the target requirement text is determined based on the characteristics of the target requirement text; as well as The updated target requirement text features are processed using the aforementioned large visual model to obtain an updated target image, wherein the updated target requirement text features are determined based on the updated target requirement text.

4. The method according to claim 1, wherein, The image attributes include at least one of the following: Style attributes, texture attributes, font attributes, preset style attributes.

5. A method for training a large visual model, comprising: The sample target demand text features, sample noise images, and label images are obtained. The sample target demand text features are determined by processing the sample demand text using a large language model. The sample target demand text features include sample image word features and sample attribute hint features. The sample image word features represent the label image words displayed in the label image. The attribute hint features are related to the label image attributes of the label image words. The sample noise image is determined by adding noise to the label image. The initial visual large model is used to process the sample target demand text features and sample noise image features to obtain the sample target image. The sample target image includes sample image character regions related to the label image characters. The sample noise image features are determined based on the sample noise image. as well as The initial visual large model is trained based on the sample target image and the label image to obtain the trained visual large model; wherein, the initial visual large model includes N cascaded initial image denoising networks, where N>1 and N is an integer; The step of processing the target requirement text features and sample noise image features using the initial large visual model to obtain the sample target image includes: Based on the attention mechanism, the N cascaded initial image denoising networks are used to process the sample target demand text features and the sample noise features to obtain the sample target image; The large visual model is used to determine the target image by performing the following operations: Based on preset noise information, spatial domain modulation processing is performed on the intermediate text fusion feature of level n-1 and the noise feature to obtain the nth spatial domain text feature and the nth spatial domain image feature corresponding to the preset spatial domain, where N≥n>1 and N and n are integers; Based on the attention mechanism, the text features and image features of the nth spatial domain are fused to obtain the text fusion features and image fusion features of the nth spatial domain; Based on preset noise information, the nth spatial domain text fusion feature and the nth spatial domain image fusion feature are spatially modulated to obtain the nth level intermediate text fusion feature and the nth level intermediate image fusion feature output by the nth image denoising network; When N=n, the target image is determined based on the Nth level intermediate image fusion features.

6. The method according to claim 5, wherein, The initial large visual model was determined based on the following operations: Obtain a pre-trained visual model, wherein the pre-trained visual model includes K pre-trained image denoising networks, where K is an integer greater than 1; At least one of the pre-trained image denoising networks in the pre-trained visual model is copied to obtain an extended image denoising network; and The pre-trained visual model is updated based on at least one of the extended image denoising networks to obtain an initial large visual model with N initial image denoising networks, where N > K.

7. The method according to claim 6, wherein, The i-th initial image denoising network of the initial large visual model is the j-th pre-trained image denoising network of the pre-trained visual model, and the (i+1)-th initial image denoising network of the initial large visual model is the extended image denoising network corresponding to the j-th pre-trained image denoising network, where i and j are integers.

8. The method according to claim 6, wherein, The initial visual large model includes a cascaded initial image denoising module i and an initial image denoising module (i+1); Wherein, the multiple initial image denoising networks in the i-th initial image denoising module are multiple pre-trained image denoising networks of the pre-trained visual model, and the multiple initial image denoising networks in the (i+1)-th initial image denoising module are multiple extended image denoising networks; the multiple extended image denoising networks are determined by copying the network structure of the multiple pre-trained image denoising networks.

9. The method according to claim 5, wherein, The large visual model comprises N cascaded image denoising networks, where N > 1 and N is an integer; The method further includes: The trained large visual model is pruned to obtain a pruned large visual model; and Based on the trained visual big model, knowledge distillation is performed on the pruned visual big model to obtain an updated visual big model.

10. The method according to claim 9, wherein, The pruned visual large model includes M cascaded pruned image denoising networks, where N > M > 1, and M is an integer; The step of performing knowledge distillation on the pruned visual large model based on the trained visual large model includes: The trained visual large model is used to process the sample scene requirement text features and sample scene noise image features related to the specified scene, so as to obtain the Nth level intermediate scene image fusion feature and Nth level intermediate scene text fusion feature output by the Nth image denoising network. The pruned visual big model is used to process the sample scene requirement text features and the sample scene noise image features to obtain the Mth level intermediate pruned image fusion features and the Mth level intermediate pruned text fusion features output by the Mth pruned image denoising network. The knowledge distillation loss is determined based on the Nth-level intermediate scene image fusion features and the Nth-level intermediate scene text fusion features, as well as the Mth-level intermediate pruned image fusion features and the Mth-level intermediate pruned text fusion features; and The model parameters of the Mth pruned image denoising network are adjusted based on the knowledge distillation loss.

11. An image generation device based on a large language model, comprising: The first acquisition module is used to acquire the requirement text; The first processing module is used to process the requirement text using a large language model to obtain target requirement text features. The target requirement text features include image word features and attribute hint features. The image word features represent the image words that need to be displayed in the generated target image. The attribute hint features are related to the image attributes of the image words. The second processing module is used to process the target requirement text features using a large visual model to obtain the target image. The target image includes image word regions displayed based on the image attributes, and the image word regions represent image words in the requirement text. The large visual model includes N cascaded image denoising networks, where N > 1 and N is an integer. The second processing module further includes: The first image denoising processing submodule is used to process the target text features and preset noise features based on the attention mechanism using the cascaded N image denoising networks to obtain the target image; The first image noise reduction processing submodule includes a first processing unit and a second processing unit, wherein the first processing unit includes: The first spatial domain modulation subunit is used to perform spatial domain modulation processing on the (n-1)th level intermediate text fusion feature and the noise feature based on preset noise information to obtain the nth spatial domain text feature and the nth spatial domain image feature corresponding to the preset spatial domain. The feature fusion subunit is used to fuse the text features and image features in the nth spatial domain based on an attention mechanism to obtain the text fusion features and image fusion features in the nth spatial domain; and The second spatial domain modulation subunit is used to perform spatial domain modulation on the nth spatial domain text fusion feature and the nth spatial domain image fusion feature based on preset noise information, so as to obtain the nth level intermediate text fusion feature and the nth level intermediate image fusion feature output by the nth image denoising network; The second processing unit is used to determine the target image based on the Nth level intermediate image fusion features when N=n.

12. The apparatus according to claim 11, wherein, The first-level intermediate text fusion features and the first-level intermediate image fusion features are determined by the first image denoising network processing the target text features and the noise features.

13. The apparatus according to claim 11, wherein, Also includes: An update module is used to update the target requirement text in response to a text editing request, thereby obtaining the updated target requirement text, wherein the target requirement text is determined based on the characteristics of the target requirement text; as well as The third processing module is used to process the updated target demand text features using the visual large model to obtain an updated target image, wherein the updated target demand text features are determined based on the updated target demand text.

14. The apparatus according to claim 11, wherein, The image attributes include at least one of the following: Style attributes, texture attributes, font attributes, preset style attributes.

15. A training device for a large visual model, comprising: The second acquisition module is used to acquire sample target demand text features, sample noise images, and label images. The sample target demand text features are determined by processing the sample demand text using a large language model. The sample target demand text features include sample image character features and sample attribute hint features. The sample image character features represent the label image characters displayed in the label image. The attribute hint features are related to the label image attributes of the label image characters. The sample noise image is determined by adding noise to the label image. The fourth processing module is used to process the sample target demand text features and sample noise image features using the initial visual large model to obtain a sample target image. The sample target image includes sample image character regions related to the label image characters. The sample noise image features are determined based on the sample noise image. as well as The training module is used to train the initial large visual model based on the sample target image and the label image to obtain the trained large visual model; wherein, the initial large visual model includes N cascaded initial image denoising networks; The fourth processing module includes: The second image denoising processing submodule is used to process the sample target demand text features and the sample noise features based on the attention mechanism using the cascaded N initial image denoising networks to obtain the sample target image; The large visual model is used to determine the target image by performing the following operations: Based on preset noise information, spatial domain modulation processing is performed on the intermediate text fusion feature of level n-1 and the noise feature to obtain the nth spatial domain text feature and the nth spatial domain image feature corresponding to the preset spatial domain, where N≥n>1 and N and n are integers; Based on the attention mechanism, the text features and image features of the nth spatial domain are fused to obtain the text fusion features and image fusion features of the nth spatial domain; Based on preset noise information, the nth spatial domain text fusion feature and the nth spatial domain image fusion feature are spatially modulated to obtain the nth level intermediate text fusion feature and the nth level intermediate image fusion feature output by the nth image denoising network; When N=n, the target image is determined based on the Nth level intermediate image fusion features.

16. The apparatus according to claim 15, wherein, The initial large visual model was determined based on the following operations: Obtain a pre-trained visual model, wherein the pre-trained visual model includes K pre-trained image denoising networks, where K is an integer greater than 1; At least one of the pre-trained image denoising networks in the pre-trained visual model is copied to obtain an extended image denoising network; and The pre-trained visual model is updated based on at least one of the extended image denoising networks to obtain an initial large visual model with N initial image denoising networks, where N > K.

17. The apparatus according to claim 16, wherein, The i-th initial image denoising network of the initial large visual model is the j-th pre-trained image denoising network of the pre-trained visual model, and the (i+1)-th initial image denoising network of the initial large visual model is the extended image denoising network corresponding to the j-th pre-trained image denoising network, where i and j are integers.

18. The apparatus according to claim 16, wherein, The initial visual large model includes a cascaded initial image denoising module i and an initial image denoising module (i+1); Wherein, the multiple initial image denoising networks in the i-th initial image denoising module are multiple pre-trained image denoising networks of the pre-trained visual model, and the multiple initial image denoising networks in the (i+1)-th initial image denoising module are multiple extended image denoising networks; the multiple extended image denoising networks are determined by copying the network structure of the multiple pre-trained image denoising networks.

19. The apparatus according to claim 15, wherein, The large visual model comprises N cascaded image denoising networks; The device further includes: The pruning module is used to prune the trained large visual model to obtain a pruned large visual model; and The knowledge distillation module is used to perform knowledge distillation on the pruned visual large model based on the trained visual large model, so as to obtain an updated visual large model.

20. The apparatus according to claim 19, wherein, The pruned visual large model includes M cascaded pruned image denoising networks, where N > M > 1, and M is an integer; The knowledge distillation module includes: The first processing submodule is used to process the sample scene requirement text features and sample scene noise image features related to the specified scene using the trained visual large model, and to obtain the Nth level intermediate scene image fusion features and Nth level intermediate scene text fusion features output by the Nth image denoising network. The second processing submodule is used to process the sample scene requirement text features and the sample scene noise image features using the pruning visual big model to obtain the Mth level intermediate pruning image fusion feature and the Mth level intermediate pruning text fusion feature output by the Mth pruning image denoising network. The loss determination submodule is used to determine the knowledge distillation loss based on the Nth-level intermediate scene image fusion features and the Nth-level intermediate scene text fusion features, as well as the Mth-level intermediate pruned image fusion features and the Mth-level intermediate pruned text fusion features; and The training submodule is used to adjust the model parameters of the Mth pruned image denoising network based on the knowledge distillation loss.

21. An artificial intelligence agent system, comprising: The input module is used to receive input information; The processing module is configured to determine a target task based on the input information received by the input module, determine a large language model or a large visual model or an initial large visual model based on the target task, and execute the method of any one of claims 1 to 4 by calling the large language model and / or the large visual model, or execute the method of any one of claims 5 to 10 by calling the initial large visual model to obtain output information; An output module is used to output the output information obtained by the processing module.

22. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

23. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

24. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images