Controllable image generation method based on semantic embedding and preference modeling

Abstract

The invention discloses a controllable image generation method based on semantic embedding and preference modeling. The controllable image generation method comprises the following steps of S1, finely adjusting a large language model of a data set by using an image-text; wherein the image-text pair data set comprises a plurality of triads, and each triad comprises an input image, an instruction text and a generated image; s2, optimizing instruction text embedding by utilizing the fine-tuned large language model and image reconstruction; s3, embedding the optimized instruction text into a fine tuning pre-training image to generate a model; s4, optimizing the fine-tuned pre-training image generation model by using a preference model; wherein the preference model is obtained based on training of a generated image marked with ranking data; and S5, generating a model by using the optimized pre-training image, and obtaining a generated image matched with the instruction text to be processed and the input image to be processed. The image generated by the method provided by the invention better conforms to the aesthetic appreciation and value of human beings, and the high-quality image associated with the input image and the instruction text can be stably generated under various noise conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and more particularly to a controllable image generation method based on semantic embedding and preference modeling. Background Art

[0002] As the demand for visual content in digital media, advertising, and scientific research continues to grow, controllable image generation technology has become increasingly important. However, existing technologies face challenges in achieving precise control and preference consistency over the image generation process. Studies have shown that the quality of images generated by controllable image generation models is closely related to instruction text embeddings and preference data. This means that in order to generate images that better meet user expectations, it is necessary to more accurately understand and simulate user instructions and preferences, as well as more effectively incorporate this information into the image generation process. This involves not only technical innovations, such as using large language models to optimize text embeddings, but also in-depth research and modeling of user preferences, as well as the development of more effective evaluation and optimization mechanisms. Through these efforts, the accuracy and personalization of image generation can be improved, thereby better meeting user needs and enhancing user experience.

[0003] In conventional technology, in order to solve the problem of aligning the generated image with the instruction text, the only way is to adjust the degree of noise added to the model and the length of the model denoising process to achieve the alignment of the instruction text of the generated image with the image text of the generated image. At the same time, due to the subjectivity and diversity of human preferences, this process has limitations. Although existing methods for evaluating the quality of generated images, such as CLIP Score and FID, can express the quality of images to a certain extent, they are not sufficient to fully capture the real perception and preference of humans for images.

[0004] In summary, controllable image generation is a hot area of ​​interest. For semantic embedding and preference modeling problems, there is no solution that takes into account the subjectivity and diversity of human preferences as well as precise control during image generation.

[0005] Therefore, how to provide a controllable image generation method that takes into account the subjectivity and diversity of human preferences as well as precise control during the image generation process is an issue that those skilled in the art urgently need to solve. Summary of the invention

[0006] In view of this, the object of the present invention is to provide a controllable image generation method based on semantic embedding and preference modeling.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A controllable image generation method based on semantic embedding and preference modeling comprises the following steps:

[0009] S1: fine-tuning a large language model using an image-text pair dataset; wherein the image-text pair dataset includes a plurality of triplets, each of which includes an input image, an instruction text, and a generated image;

[0010] S2: Optimize instruction text embedding using fine-tuned large language model and image reconstruction;

[0011] S3: Fine-tune the pre-trained image generation model using the optimized instruction text embedding;

[0012] S4: optimizing the fine-tuned pre-trained image generation model using the preference model; wherein the preference model is obtained by training the generated images annotated with ranking data;

[0013] S5: Using the optimized pre-trained image generation model, obtain a generated image that matches the instruction text to be processed and the input image to be processed.

[0014] Preferably, S1 specifically includes:

[0015] S11: Input the input image and the generated image into the CLIP model to obtain the input image text and the generated image text;

[0016] S12: Taking the input image text and instruction text as standard input, generating the image text as standard output to fine-tune the large language model; wherein the large language model is a pre-trained large language model with a low-rank layer added.

[0017] Preferably, S2 specifically includes:

[0018] S21: Input the input image to the CLIP model to obtain the input image text;

[0019] S22: inputting the instruction text and the input image text into the fine-tuned large language model to obtain an optimized instruction text;

[0020] S23: Input the optimized instruction text into the text encoder to obtain the instruction text embedding e tgt ;

[0021] S24: embed the input image and instruction text into e tgt Input into the pre-trained image generation model to obtain the corresponding generated image;

[0022] S25: Minimize the difference between the generated image and the input image obtained in S24 to optimize the output of the text encoder and obtain the optimized instruction text embedding e opt .

[0023] Preferably, in S24: firstly, noise is added to the input image to make it into complete Gaussian noise; and then denoising is performed step by step to restore the input image; the input image with added noise is the input image containing noise during the denoising process.

[0024] Preferably, S25 optimizes the output of the text encoder based on the following formula:

[0025]

[0026] in, represents the reconstruction loss function; x represents the input image; e represents the optimized instruction text; θ represents the parameters of the pre-trained image generation model; represents the expected value of t and ∈; ∈ represents the random noise drawn from the standard normal distribution; ∈ θ represents a pre-trained image generation model with parameters θ; x t represents the input image with added noise; t represents the time step used to control the variation of random noise ∈.

[0027] Preferably, S3 specifically includes:

[0028] Embed the optimized instruction text into e opt The input image with added noise is input into a pre-trained image generation model; and the pre-trained image generation model is fine-tuned using the reconstruction loss function.

[0029] Preferably, the preference model comprises a visual encoder, a first linear layer, a text encoder and a second linear layer;

[0030] In the preference model, the input end of the visual encoder is used to input the generated image annotated with ranking data; the output end of the visual encoder is connected to the cross-attention layer in the text encoder through the first linear layer; the input end of the text encoder is used to input the instruction text, and the output end of the text encoder is connected to the input end of the second linear layer; wherein, the input end of the visual encoder and the input end of the text encoder are the input end of the preference model; the output end of the second linear layer is the output end of the preference model.

[0031] Preferably, the preference model is obtained by training based on the following loss function:

[0032]

[0033] Among them, loss(θ') represents the loss function of the preference model; E represents the expected value of the entire data set D; D represents the data set; x represents the input image, e' represents the instruction text, y i and jrepresents two different generated images obtained under the same input image x and instruction text e'; log represents the logarithmic operator; σ represents the sigmoid function; θ' represents the parameters of the preference model; ∈ θ’ represents the preference model with parameter θ'; ∈ θ’ (x,e',θ',y i ) represents the generated image y of the input image x and the instruction text e' i The preference value of θ’ (x,e',θ',y j ) represents the generated image y of the input image x and the instruction text e' j preference value.

[0034] Preferably, S4 specifically includes:

[0035] S41: Embed e in the instruction text tgt and the optimized instruction text is embedded in e opt Perform linear interpolation between them to obtain interpolation instruction text embedding;

[0036] S42: inputting the interpolation instruction text embedding and Gaussian noise into the fine-tuned pre-trained image generation model to obtain a generated image corresponding to the interpolation instruction text embedding and Gaussian noise;

[0037] S43: embedding the interpolation instruction text and the generated image corresponding to the Gaussian noise into the preference model to obtain the corresponding preference value;

[0038] S44: Use the preference value obtained in S43 to update the parameters of the fine-tuned pre-trained image generation model to obtain an optimized pre-trained image generation model.

[0039] Preferably, S5 specifically includes:

[0040] S51: Input the input image to be processed into the CLIP model to obtain the text corresponding to the input image to be processed;

[0041] S52: inputting the text corresponding to the input image to be processed and the instruction text to be processed into the fine-tuned large language model to obtain the corresponding optimized instruction text;

[0042] S53: inputting the optimization instruction text obtained in S52 into the text encoder after output optimization to obtain the corresponding optimization instruction text embedding;

[0043] S54: embedding the optimized instruction text obtained in S53 and the input image to be processed into the optimized pre-trained image generation model to obtain a generated image that matches the instruction text to be processed and the input image to be processed.

[0044] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a controllable image generation method based on semantic embedding and preference modeling. The images generated by the method of the present invention are not only more in line with human aesthetics and values, but also can stably generate high-quality input images and images associated with instruction texts under various noise conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0046] Figure 1 A flowchart of a controllable image generation method based on semantic embedding and preference modeling provided by the present invention;

[0047] Figure 2 A schematic diagram of fine-tuning a large language model using images and texts on a data set provided in an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of optimizing instruction text embedding using a fine-tuned large language model and image reconstruction provided in an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of fine-tuning a pre-trained image generation model using optimized instruction text embedding provided in an embodiment of the present invention;

[0050] Figure 5 A schematic diagram of preference data annotated by a human evaluation expert provided in an embodiment of the present invention;

[0051] Figure 6 A schematic diagram of the structure of a preference model provided in an embodiment of the present invention;

[0052] Figure 7 A schematic diagram of a pre-trained image generation model after optimization and fine-tuning using a preference model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] like Figure 1-Figure 7 As shown, an embodiment of the present invention discloses a controllable image generation method based on semantic embedding and preference modeling, comprising the following steps:

[0055] S1: fine-tuning a large language model using an image-text pair dataset; wherein the image-text pair dataset includes a plurality of triplets, each of which includes an input image, an instruction text, and a generated image;

[0056] Specifically: Each triplet contains an input image, the corresponding instruction text, and 2 to 8 generated images that match the instruction text and the input image.

[0057] S2: Optimize instruction text embedding using fine-tuned large language model and image reconstruction;

[0058] S3: Fine-tune the pre-trained image generation model using the optimized instruction text embedding;

[0059] S4: optimizing the fine-tuned pre-trained image generation model using the preference model; wherein the preference model is obtained by training the generated images annotated with ranking data;

[0060] S5: Using the optimized pre-trained image generation model, obtain a generated image that matches the instruction text to be processed and the input image to be processed.

[0061] Furthermore, S1 specifically includes:

[0062] S11: Input the input image and the generated image into the CLIP model to obtain the input image text and the generated image text;

[0063] It can be understood that: the CLIP model is used to unify the input image and the generated image into the same modality as the instruction text;

[0064] S12: Taking the input image text and instruction text as standard input, generating the image text as standard output to fine-tune the large language model; wherein the large language model is a pre-trained large language model with a low-rank layer added.

[0065] It can be understood that: the present invention uses the input image text and instruction text as standard input and generates the image text as standard output, so that the large language model can better understand the relationship between the image content and the instruction text. The fine-tuned large language model receives the input image text and the instruction text as input, and outputs a generated image text that matches the input conditions as the optimized instruction text.

[0066] Furthermore, S2 specifically includes:

[0067] S21: Input the input image to the CLIP model to obtain the input image text;

[0068] S22: inputting the instruction text and the input image text into the fine-tuned large language model to obtain an optimized instruction text;

[0069] S23: Input the optimized instruction text into the text encoder to obtain the instruction text embedding e tgt ;

[0070] S24: embed the input image and instruction text into e tgt Input into the pre-trained image generation model to obtain the corresponding generated image;

[0071] S25: Minimize the difference between the generated image and the input image obtained in S24 to optimize the output of the text encoder and obtain the optimized instruction text embedding e opt .

[0072] It is understandable that in the process of optimizing instruction text embedding using the fine-tuned large language model and image reconstruction, the parameters of the pre-trained large language model and CLIP model are frozen.

[0073] Further, in S24: firstly, noise is added to the input image to make it into complete Gaussian noise; and then, denoising is performed step by step to restore the input image; the input image with added noise is the input image containing noise during the denoising process.

[0074] It can be understood that: the generation process of the simulated image is achieved by adding noise to the input image until it becomes a complete Gaussian noise, and then restoring the input image through step-by-step iterative denoising;

[0075] Furthermore, S25 optimizes the output of the text encoder based on the following formula:

[0076]

[0077] in, represents the reconstruction loss function; x represents the input image; e represents the optimized instruction text; θ represents the parameters of the pre-trained image generation model; represents the expected value of t and ∈; ∈ represents the random noise drawn from the standard normal distribution; ∈ θ represents a pre-trained image generation model with parameters θ; x t represents the input image with added noise; t represents the time step used to control the variation of random noise ∈.

[0078] Furthermore, S3 specifically includes:

[0079] Embed the optimized instruction text into e optThe input image with added noise is input into a pre-trained image generation model; and the pre-trained image generation model is fine-tuned using the reconstruction loss function.

[0080] Specifically: embed the optimized instruction text into e opt As a fixed input, the image generation process is simulated by adding noise to the input image until it becomes a complete Gaussian noise, and then restoring the input image through step-by-step iterative denoising; in the process of step-by-step denoising, the parameters of the pre-trained image generation model are fine-tuned using the same reconstruction loss function as S25. By changing the parameters of the pre-trained image generation model to adapt to the optimized instruction text embedding e opt The input image under the condition better adapts to the image generated by the instruction.

[0081] The input image with added noise in S3 is an input image containing noise during the denoising process.

[0082] Further, the preference model includes a visual encoder, a first linear layer, a text encoder, and a second linear layer;

[0083] In the preference model, the input end of the visual encoder is used to input the generated image annotated with ranking data; the output end of the visual encoder is connected to the cross-attention layer in the text encoder through the first linear layer; the input end of the text encoder is used to input the instruction text, and the output end of the text encoder is connected to the input end of the second linear layer; wherein, the input end of the visual encoder and the input end of the text encoder are the input end of the preference model; the output end of the second linear layer is the output end of the preference model.

[0084] It is understandable that: in the process of human evaluation experts labeling preference data, the present invention creates a data set, which includes input images, instruction texts, and generated images. For the same set of input images and 2 to 8 images generated by instruction texts, the present invention manually ranks and labels them according to the three criteria of "consistency between input images and generated images", "consistency between instruction texts and generated images", and "preference of human evaluation experts for generated images" (e.g. Figure 4 As shown in Figure 2, human preference information is incorporated into the data set. To ensure the quality of data annotation, the present invention introduces an automatic quality assessment algorithm to remove low-quality data. The criteria for removal include unclear instruction texts and image data with problems such as "violence" or "low resolution".

[0085] Since the dataset combines the input image and instruction text with 2 to 8 corresponding generated images into a unit, it is often difficult to define the quality of the sample, and using ranking can more effectively express human preferences. In this evaluation and ranking annotation process, generated images that meet the standards will receive higher rankings.

[0086] Finally, after the human evaluation of the expert preference annotations, a quality control process is introduced. In this process, human reviewers will check one by one whether the data annotations accurately reflect the real situation of the data, that is, whether the image annotations are consistent with the image content. If there are errors or inconsistencies in the annotations, the reviewers will correct them. After the review is completed, the reviewer will provide final approval to confirm that the quality of the dataset meets the requirements for training the preference model.

[0087] It can be understood that the second linear layer added in the present invention finally converts the ranking data into a preference value between -1 and 1. The more the generated image conforms to human preference, the higher the preference value. For any input image or image generated by instruction text, the preference model will assign a preference value, and the higher the value, the better the sample quality. By marking human preferences as rankings, for the same input image and instruction text, there are usually (k∈[2,8]) generated images. If there is no connection between two generated images, the maximum value is For each comparison pair, if y i Better and y j Worse, the loss function can be defined as:

[0088] Furthermore, the preference model is obtained by training based on the following loss function:

[0089]

[0090] Among them, loss(θ') represents the loss function of the preference model; E represents the expected value of the entire data set D; D represents the data set; x represents the input image, e' represents the instruction text, y i and j represents two different generated images obtained under the same input image x and instruction text e'; log represents the logarithmic operator; σ represents the sigmoid function; θ' represents the parameters of the preference model; ∈ θ’ represents the preference model with parameter θ'; ∈ θ’ (x,e',θ',y i ) represents the generated image y of the input image x and the instruction text e' i The preference value of θ’ (x,e',θ',y j ) represents the generated image y of the input image x and the instruction text e' j preference value.

[0091] Furthermore, S4 specifically includes:

[0092] S41: Embed e in the instruction text tgt and the optimized instruction text is embedded in eopt Perform linear interpolation between them to obtain interpolation instruction text embedding;

[0093] The expression embedded in the interpolation instruction text is:

[0094] represents the interpolated instruction text embedding; α is a hyperparameter used to control the degree of interpolation.

[0095] S42: inputting the interpolation instruction text embedding and Gaussian noise into the fine-tuned pre-trained image generation model to obtain a generated image corresponding to the interpolation instruction text embedding and Gaussian noise;

[0096] S43: embedding the interpolation instruction text and the generated image corresponding to the Gaussian noise into the preference model to obtain the corresponding preference value;

[0097] S44: Use the preference value obtained in S43 to update the parameters of the fine-tuned pre-trained image generation model to obtain an optimized pre-trained image generation model.

[0098] It can be understood that: the preference model denoising iteration number t of the present invention is set between 30 and 40, that is, under the premise that the interpolation instruction text embedding is fixed, the denoising iteration number of the pre-trained image generation model is randomly selected, and its value is between 30-40. During the training process, after each t denoising iterations are executed, the pre-trained image generation model outputs a corresponding generated image, and each time the pre-trained image generation model outputs a generated image, the generated image is input into the preference model to obtain the preference value, and the preference value is used as the loss value of the pre-trained image generation model to update the parameters of the pre-trained image generation model; the above parameter update process is repeated until all interpolation instruction text embedding applications are completed, and the optimized pre-trained image generation model is obtained.

[0099] The final loss function of the present invention is as follows:

[0100]

[0101] Among them, L final represents the final loss function; λ represents a hyperparameter, which is used to control the complexity of the model and prevent overfitting; E represents the expected value, represents the instruction text embedding after the i-th interpolation; x i represents the i-th input image; φ represents the mapping function from preference value to loss; r represents the preference model; Denotes a pre-trained image generation model with parameter θ using The image generated as input condition.

[0102] Furthermore, S5 specifically includes:

[0103] S51: Input the input image to be processed into the CLIP model to obtain the text corresponding to the input image to be processed;

[0104] S52: inputting the text corresponding to the input image to be processed and the instruction text to be processed into the fine-tuned large language model to obtain the corresponding optimized instruction text;

[0105] S53: inputting the optimization instruction text obtained in S52 into the text encoder after output optimization to obtain the corresponding optimization instruction text embedding;

[0106] S54: embedding the optimized instruction text obtained in S53 and the input image to be processed into the optimized pre-trained image generation model to obtain a generated image that matches the instruction text to be processed and the input image to be processed.

[0107] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0108] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A controllable image generation method based on semantic embedding and preference modeling, characterized in that: The following steps are involved: S1: fine-tuning a large language model using an image-text pair dataset; wherein the image-text pair dataset includes a plurality of triplets, each of which includes an input image, an instruction text, and a generated image; S2: Optimize instruction text embedding using fine-tuned large language model and image reconstruction; S3: Fine-tune the pre-trained image generation model using the optimized instruction text embedding; S4: optimizing the fine-tuned pre-trained image generation model using the preference model; wherein the preference model is obtained by training the generated images annotated with ranking data; S5: Using the optimized pre-trained image generation model, obtain a generated image that matches the instruction text to be processed and the input image to be processed.

2. The controllable image generation method based on semantic embedding and preference modeling according to claim 1, characterized in that: S1 specifically includes: S11: Input the input image and the generated image into the CLIP model to obtain the input image text and the generated image text; S12: Taking the input image text and instruction text as standard input, generating the image text as standard output to fine-tune the large language model; wherein the large language model is a pre-trained large language model with a low-rank layer added.

3. The controllable image generation method based on semantic embedding and preference modeling according to claim 1, characterized in that: S2 specifically includes: S21: Input the input image to the CLIP model to obtain the input image text; S22: inputting the instruction text and the input image text into the fine-tuned large language model to obtain an optimized instruction text; S23: Input the optimized instruction text into the text encoder to obtain the instruction text embedding e tgt ; S24: embed the input image and instruction text into e tgt Input into the pre-trained image generation model to obtain the corresponding generated image; S25: Minimize the difference between the generated image and the input image obtained in S24 to optimize the output of the text encoder and obtain the optimized instruction text embedding e opt .

4. The controllable image generation method based on semantic embedding and preference modeling according to claim 3, characterized in that: In S24: firstly, noise is added to the input image to make it into complete Gaussian noise; and then denoising is performed step by step to restore the input image; the input image with added noise is the input image containing noise during the denoising process.

5. The controllable image generation method based on semantic embedding and preference modeling according to claim 3, characterized in that: S25 optimizes the output of the text encoder based on the following formula: in, represents the reconstruction loss function; x represents the input image; e represents the optimized instruction text; θ represents the parameters of the pre-trained image generation model; represents the expected value of t and e; e represents the random noise drawn from the standard normal distribution; ε θ represents a pre-trained image generation model with parameters θ; x t represents the input image with added noise; t represents the time step used to control the change of random noise e.

6. The controllable image generation method based on semantic embedding and preference modeling according to claim 5, characterized in that: S3 specifically includes: Embed the optimized instruction text into e opt The input image with added noise is input into a pre-trained image generation model; and the pre-trained image generation model is fine-tuned using the reconstruction loss function.

7. The controllable image generation method based on semantic embedding and preference modeling according to claim 1, characterized in that: The preference model includes a visual encoder, a first linear layer, a text encoder, and a second linear layer; In the preference model, the input end of the visual encoder is used to input the generated image annotated with ranking data; the output end of the visual encoder is connected to the cross-attention layer in the text encoder through the first linear layer; the input end of the text encoder is used to input the instruction text, and the output end of the text encoder is connected to the input end of the second linear layer; wherein, the input end of the visual encoder and the input end of the text encoder are the input end of the preference model; the output end of the second linear layer is the output end of the preference model.

8. The controllable image generation method based on semantic embedding and preference modeling according to claim 7, characterized in that: The preference model is trained based on the following loss function: Among them, loss(θ') represents the loss function of the preference model; E represents the expected value of the entire data set D; D represents the data set; x represents the input image, e' represents the instruction text, y i and j represents two different generated images obtained under the same input image x and instruction text e'; log represents the logarithmic operator; σ represents the sigmoid function; θ' represents the parameters of the preference model; ∈ θ’ represents the preference model with parameter θ'; ∈ θ’ (x, e', θ', y i ) represents the generated image y of the input image x and the instruction text e' i The preference value of ∈ θ’ (x, e', θ', y j ) represents the generated image y of the input image x and the instruction text e' j preference value.

9. The controllable image generation method based on semantic embedding and preference modeling according to claim 7, characterized in that: S4 specifically includes: S41: Embed e in the instruction text tgt and the optimized instruction text is embedded in e opt Perform linear interpolation between them to obtain interpolation instruction text embedding; S42: inputting the interpolation instruction text embedding and Gaussian noise into the fine-tuned pre-trained image generation model to obtain a generated image corresponding to the interpolation instruction text embedding and Gaussian noise; S43: inputting the generated image corresponding to the interpolation instruction text embedding and Gaussian noise into the preference model to obtain the corresponding preference value; S44: Use the preference value obtained in S43 to update the parameters of the fine-tuned pre-trained image generation model to obtain an optimized pre-trained image generation model.

10. The controllable image generation method based on semantic embedding and preference modeling according to claim 1, characterized in that: S5 specifically includes: S51: Input the input image to be processed into the CLIP model to obtain the text corresponding to the input image to be processed; S52: Inputting the text corresponding to the input image to be processed and the instruction text to be processed into the fine-tuned large language model to obtain the corresponding optimized instruction text; S53: inputting the optimization instruction text obtained in S52 into the text encoder after output optimization to obtain the corresponding optimization instruction text embedding; S54: embedding the optimized instruction text obtained in S53 and the input image to be processed into the optimized pre-trained image generation model to obtain a generated image that matches the instruction text to be processed and the input image to be processed.

Images