A method and system for optimizing large models of Wensheng graphs based on multi-objective optimization

Abstract

The present invention relates to the technical field of model optimization, and in particular to a method and system for optimizing a large model of a Wensheng graph based on multi-objective optimization. The method comprises: acquiring text data; constructing a large model of a Wensheng graph based on a diffusion model; constructing a multi-objective optimization problem based on optimization variables, optimizing parameters of the large model of the Wensheng graph based on the multi-objective optimization problem and using the NSGA‑II algorithm; based on parameter optimization, using the large model of the Wensheng graph to perform reverse diffusion to generate a final user demand image. The Wensheng graph optimization fine-tuning method proposed in the present invention can optimize multiple conflicting objectives at the same time. For the fine-tuning of the Wensheng graph model, the clarity, color fidelity and detail expression of the image can be improved at the same time. At the same time, it is ensured that the generated image is highly consistent with the input text in semantics.

Description

Technical Field

[0001] The present invention relates to the technical field of model optimization, and in particular to a method and system for optimizing a large model of a Vincent graph based on multi-objective optimization. Background Art

[0002] With the development of generative models such as diffusion models (SD) and variational autoencoders (VAEs), significant progress has been made in the field of text-to-image synthesis. These technologies enable computers to generate corresponding images based on input natural language descriptions, thus having broad application prospects in art creation, advertising design, game development and other fields. However, existing large text-to-image models usually need to deal with complex optimization problems in multiple dimensions to generate high-quality images that conform to text descriptions.

[0003] In practical applications, the generated images not only need to conform to the semantic information of the text description, but also need to perform well in terms of visual quality, content diversity, etc. However, existing text-generated image models often have shortcomings in this regard. For example, the quality of the generated images may not be high enough, or the generated content may not be completely consistent with the text description. The existence of these problems means that the model needs further optimization and fine-tuning to improve the overall performance of the generated images.

[0004] NSGA-II is a multi-objective optimization algorithm that can optimize multiple objective functions at the same time and find the global optimal solution through an evolutionary process. By applying NSGA-II to the fine-tuning process of the text-generated graph model, the quality and diversity of the generated images can be optimized while maintaining the consistency of the text description, thereby achieving better generation effects.

[0005] This fine-tuning method can be widely used in scenarios that require high-quality image generation, such as personalized advertising creation, virtual scene generation, and automated design. By combining the NSGA-II optimization algorithm, the Wensheng Graph model can better adapt to different application requirements and provide higher generated image quality and richer content expression.

[0006] Therefore, there is an urgent need for a method that combines the NSGA-II algorithm and the Wensheng graph model. Summary of the invention

[0007] In order to solve the above-mentioned problems, the present invention provides a method and system for optimizing a large model of a Wensheng graph based on multi-objective optimization.

[0008] In the first aspect, the present invention provides a method for optimizing a large model of a Wensheng graph based on multi-objective optimization, which adopts the following technical solution:

[0009] A method for optimizing a large model of a Wensheng graph based on multi-objective optimization, comprising:

[0010] Get text data;

[0011] Construct a large model of the Vincent graph based on the diffusion model;

[0012] Construct a multi-objective optimization problem based on the optimization variables, and optimize the parameters of the Wensheng graph model based on the multi-objective optimization problem and using the NSGA-II algorithm;

[0013] Based on parameter optimization, the Wensheng graph model is used for back diffusion to generate the final user demand image.

[0014] Furthermore, the constructing of the large model of the Wensheng graph based on the diffusion model includes constructing the large model of the Wensheng graph based on the scalable model architecture of the diffusion model, wherein the reverse generation process of the diffusion model is:

[0015]

[0016] The reverse SDE when the condition is generated is:

[0017]

[0018] Using the Bayesian formula to derive SDE, it can be expressed as:

[0019]

[0020] in, The derivative gradient is 0. Substituting the above formula into the conditions to generate SDE, we can get:

[0021]

[0022] in, It is the probability of condition y with x as the condition and time t. It is a classification neural network.

[0023] Furthermore, the multi-objective optimization problem is constructed based on the optimization variables, including establishing an optimization problem for maximizing the image quality of the Wensheng image. The goal is to generate an image with high perceptual quality by adjusting the model input and the generation mechanism, which is expressed as:

[0024]

[0025] Among them, image perceptual quality Q, latent variable z, text description t, describes image x.

[0026] Furthermore, the multi-objective optimization problem constructed based on the optimization variables also includes an optimization problem based on maximum semantic consistency, where the goal is to make the text graph and the input text description semantically consistent, which can be expressed as:

[0027]

[0028] Among them, S(x,t,z) represents the semantic consistency score of the generated image x and text description t, and z is the latent variable.

[0029] Furthermore, the multi-objective optimization problem constructed based on the optimization variables also includes establishing an optimization problem for maximizing content diversity, the goal of which is to generate a set of samples with diverse content, expressed as:

[0030]

[0031] Among them, D(X) is the diversity measure of the generated sample set X={x1,x2,…,xn}.

[0032] Furthermore, the method is based on a multi-objective optimization problem and uses the NSGA-II algorithm to optimize the parameters of the diffusion model, including using the NSGA-II algorithm to optimize the parameters of the diffusion model with the optimization objectives of maximizing image quality, semantic consistency, and content diversity, and combining the output solution of the optimization algorithm to train the diffusion model to optimize the generation quality of the image.

[0033] Furthermore, the method based on parameter optimization and using the Wensheng graph model to perform back diffusion to generate the final user required image includes using a noise predictor to estimate the noise of the image in the latent space and gradually removing the noise to generate a clear image, which is described as:

[0034]

[0035] where xt is the image at time step t, θ(xt,t) is the noise estimate of the noise predictor for the current image xt and time step t.

[0036] In the second aspect, a large model optimization system of Wensheng graph based on multi-objective optimization includes:

[0037] The data acquisition module is configured to acquire text data;

[0038] The model building module is configured to build a large model of the Wensheng graph based on the diffusion model;

[0039] The optimization module is configured to construct a multi-objective optimization problem based on the optimization variables, and optimize the parameters of the Wensheng graph model based on the multi-objective optimization problem and using the NSGA-II algorithm;

[0040] The reverse diffusion module is configured to generate the final user demand image by reverse diffusion based on parameter optimization using the Wensheng graph model.

[0041] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a method for optimizing a large model of a Vincent graph based on multi-objective optimization.

[0042] In a fourth aspect, the present invention provides a terminal device, including a processor and a computer-readable storage medium, the processor being used to implement various instructions; the computer-readable storage medium being used to store multiple instructions, the instructions being suitable for being loaded by the processor and executing the method for optimizing a large model of a Vincent graph based on multi-objective optimization.

[0043] In summary, the present invention has the following beneficial technical effects:

[0044] (1) The text-based graph optimization fine-tuning method proposed in the present invention can simultaneously optimize multiple conflicting objectives. The fine-tuning of the text-based graph model can simultaneously improve the clarity, color fidelity, and detail expression of the image. At the same time, it ensures that the generated image is highly semantically consistent with the input text.

[0045] (2) The method for fine-tuning the optimization of the Vincent graph of the present invention can better balance the interests and goals of different parties by comprehensively considering and modeling multiple objectives. It can effectively distinguish different solutions and maintain diversity. For the Vincent graph model, this can help generate images of different types and styles, ensuring that the fine-tuned model has the ability to generate images of different styles and high quality. Compared with other multi-objective optimization algorithms, NSGA-II has an advantage in computational complexity by adopting a streamlined sorting and selection mechanism. For the fine-tuning of large Vincent graph models, especially when processing large-scale data, NSGA-II can complete the multi-objective optimization process with lower computing resource overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of a method for optimizing a large model of a Vincent graph based on multi-objective optimization according to Embodiment 1 of the present invention;

[0047] Figure 2 is a schematic diagram of a multi-objective optimization problem in Example 1 of the present invention;

[0048] Figure 3 is a schematic diagram of the NSGA-II optimization algorithm of Example 1 of the present invention;

[0049] Figure 4 1 is a schematic diagram comparing the image quality of the NSGA-II optimization algorithm of Example 1 of the present invention and other algorithms;

[0050] Figure 5 1 is a schematic diagram comparing the semantic consistency of the NSGA-II optimization algorithm of Example 1 of the present invention and other algorithms;

[0051] Figure 6 1 is a schematic diagram comparing the content diversity of the NSGA-II optimization algorithm of Example 1 of the present invention and other algorithms. DETAILED DESCRIPTION

[0052] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0053] Example 1

[0054] Reference Figure 1 , a method for optimizing a large model of a Wensheng graph based on multi-objective optimization in this embodiment includes:

[0055] Get text data;

[0056] Construct a large model of the Vincent graph based on the diffusion model;

[0057] Construct a multi-objective optimization problem based on the optimization variables, and optimize the parameters of the Wensheng graph model based on the multi-objective optimization problem and using the NSGA-II algorithm;

[0058] Based on parameter optimization, the Wensheng graph model is used for back diffusion to generate the final user demand image.

[0059] Specifically,

[0060] S1. Obtaining text data; including using a user's input text database and using text inversion technology to add text descriptions to images.

[0061] S2. Construct a large model of the Wensheng graph based on the diffusion model;

[0062] According to the text input of the user's image requirements, the pre-trained text-generated image model based on the diffusion model is used. At the same time, the NSGA-II algorithm is used to optimize the model parameters to ensure the stable output of the diffusion model, improve the image generation quality, and the service quality of the raw image model. Finally, the final user demand image is generated through the reverse diffusion reflection diffusion process based on the probability diffusion model. Among them, the text generation model is based on the scalable model architecture of the diffusion model. The reverse generation process of the diffusion model is:

[0063]

[0064] The reverse SDE when the condition is generated is:

[0065]

[0066] Using the Bayesian formula :

[0067]

[0068] in, The derivative gradient is 0. Substituting the above formula into the conditions to generate SDE, we can get:

[0069]

[0070] in, It is the probability of condition y with x as the condition and time t. It is a classification neural network.

[0071] S3. Multi-objective optimization problem definition,

[0072] Among them, text image generation is regarded as a multi-objective optimization problem, and multiple objective functions are set, such as image quality (measured by perceptual loss or structural similarity), semantic consistency (measured by matching the semantic similarity of text description with generated images), and semantic consistency measured by diversity indicators).

[0073] Determination of optimization variables: The fine-tunable parameters of the Wenshengtu model, including model weights and hyperparameters of the generator network, are used as optimization variables.

[0074] (1) Establish an optimization problem to maximize the perceptual quality of a text-based image. The goal is to generate images with the highest perceptual quality by adjusting the model input and generation mechanism.

[0075]

[0076] Text description t: The text input describing the image x affects the content of image generation.

[0077] Latent variable z: represents the latent features of image x (such as color, style, structure, etc.), which can be controlled by the intermediate feature vector of the generative model. Given the input text description t, we hope to find the optimal latent variable z to maximize the perceived quality Q of the generated image.

[0078] (2) Establish an optimization problem to maximize semantic consistency; the goal is to make the generated image as semantically consistent as possible with the input text description. Here, S(x, t, z) represents the semantic consistency score of the generated image x and text description t, and z is a potential variable that affects the generation process.

[0079]

[0080] Text description t: Descriptive text of the generated image, which is the input of the optimization problem. Latent variable z: Used to control the features of the generated image, such as style, content, etc., which may be the intermediate feature representation of the text-to-image model. Given the text description t, find the optimal latent variable z* to maximize the semantic consistency between the image and the text. Construct a loss function to penalize the semantic inconsistency between the image and the text description:

[0081] .

[0082] (3) Establish an optimization problem for maximizing content diversity. The optimization problem for maximizing content diversity aims to generate a set of images or texts whose content is as rich and diverse as possible while maintaining relevance to the initial input. The goal is to generate a set of content-diverse samples that both meet the input requirements (such as text descriptions) and show differentiation in terms of vision, semantics, etc.

[0083]

[0084] Among them, D(X) is the diversity measure of the generated sample set X={x1,x2,…,xn}.

[0085] Text description t: Given a text input, define the basic requirements for sample generation (such as the text-to-graph task). Latent variables z1, z2, …, z n Represents the latent features of generated images or texts. Adjusting these latent variables can control the diversity of generated content.

[0086] Given a text description t, we want to find an optimal set of latent variables {z1 * ,z2 * ,…,z n *}, the generated sample set X={x1,x2,…,x n}Has maximum diversity while still being consistent with the text description.

[0087] (4) Latent space distance: Diversity is measured by measuring the distribution differences of generated samples in the latent space. The more dispersed the distribution of the generated sample set X in the latent space, the more diverse the content.

[0088] The diversity of content is measured according to the potential space distance, and the distribution difference of the generated sample distribution space is used to improve the content diversity of the generated image, which is expressed as:

[0089] .

[0090] S4. Optimize the parameters of the diffusion model based on the multi-objective optimization problem and use the NSGA-II algorithm;

[0091] in,

[0092] (1) Generation of the initial population: At the beginning of model fine-tuning, an initial set of solutions is generated. Each solution corresponds to a set of parameters or hyperparameter settings of the text-based graph model. These solutions are initialized by random sampling or in a specific distribution (such as latent space); given a text description t, a set of optimal latent variables {z1*,z2*,…,zn*} is initialized, and the generated sample set X={x1,x2,…,xn} is generated;

[0093] (2) Objective function design: For the text-to-image task, multiple objective functions are set: Image quality: For example, it is measured by comparing the difference between the generated image and the real image. Semantic consistency: For example, using a cross-modal model such as CLIP to calculate the semantic relevance score between the input text and the generated image. Generated content diversity loss: It measures the diversity of generation by measuring the feature differences or the distance in the latent space between the generated image sets.

[0094]

[0095] (3) Construct an optimization problem. Based on the framework of the classic Wensheng graph diffusion model, use a multi-objective optimization algorithm to realize the influence of different hyperparameters on the generation results of the Wensheng graph model.

[0096] (4) Based on the established multi-objective optimization problem, the decision-making area space of parallel multi-objective optimization of multiple demands is considered, and the Pareto optimal solution set that balances the indicators of multiple demands is found in the feasible area.

[0097] (5) Use the improved NSGA-II optimization algorithm to optimize the optimization problems in complex multi-party scenarios.

[0098] Constraints: The generated image must be highly semantically consistent with the input text. This can be constrained by performing cross-modal semantic alignment between the input text and the generated image. The image P is consistent with the text set S:

[0099] .

[0100] Constraints: The generated image must meet basic quality requirements, such as clarity, color accuracy, and detail integrity. The generated image quality QoP needs to meet M quality requirement constraints:

[0101] .

[0102] Constraints: The generated images cannot be too similar and should maintain sufficient diversity to avoid the generated images concentrating on a specific style or pattern:

[0103] .

[0104] Constraints: Fine-tuning the Wensheng graph model usually requires a lot of computing resources and time, so during the optimization process, it is necessary to limit the computational complexity and training time:

[0105] .

[0106] The NSGA-II (Non-dominated Sorting Genetic Algorithm II) algorithm has shown many advantages in fine-tuning the Text-to-Image Generation model, especially in multi-objective optimization problems. It performs well in dealing with complex, multi-objective optimization problems, and the optimization of the Text-to-Image Generation model usually involves multiple conflicting objectives, such as image quality, semantic consistency, generation diversity, etc. These objectives are often in conflict (for example, while improving image quality, generation diversity may be reduced), and NSGA-II can optimize these conflicting objectives at the same time. It finds a set of solutions that perform well in each objective through the Pareto optimal solution set, not only focusing on one objective, but looking for the balance point of multiple objectives.

[0107] The specific steps of the optimization algorithm are as follows:

[0108] (1) Initialize the population, where the individuals in the population are the input parameter variables X={x1,x2,…,xn};

[0109] (2) Calculate the non-dominated sorting, that is, calculate the number of solutions dominated by each individual by other individuals and sort them;

[0110] (3) Select the parent individual Select the non-dominated solution as the parent individual ;

[0111] (4) Merge the parent F and the offspring individual O, ;

[0112] (5) Competitive selection generates elite offspring individuals ME;

[0113] (6) Recalculate the non-dominated solution from the new generation of ME individuals;

[0114] (7) Repeat steps (2) to (6);

[0115] (8) Generate the number of solutions that satisfy Pareto optimality.

[0116] NSGA-II uses a fast non-dominated sorting algorithm in each generation of evolution, which greatly reduces the complexity and makes the sorting operation more efficient. This can save a lot of time and resources for fine-tuning the Wensheng graph model, especially when large-scale data and models need to be processed. By retaining the excellent solutions of each generation, the stability and continuity of the optimization process are ensured, allowing the model to converge to the optimal solution set more quickly. For Wensheng graph model fine-tuning, this means that high-quality, semantically consistent and diverse generated images can be obtained faster.

[0117] S5. Based on the above optimization method, the diffusion model is used for reverse diffusion to realize the text-to-image function. In the SD diffusion model, the forward diffusion process gradually adds noise in the latent space, gradually transforming the image into a completely random noise distribution. This process simulates the diffusion phenomenon in physics. By gradually adding Gaussian noise, the characteristics of the image gradually disappear and eventually become pure random noise.

[0118]

[0119] In the reverse diffusion stage, the model's task is to restore a clear image from noise. The specific steps are as follows:

[0120] The model uses a noise predictor to estimate the noise of the image in the latent space and gradually removes the noise to generate a clear image. This process can be described by the following formula:

[0121]

[0122] xt is the image at time step t. θ(xt,t) is the noise estimate of the noise predictor for the current image xt and time step t.

[0123] Figure 1 The basic framework of the text graph model used in the present invention uses the classic transformer framework and the method of optimizing feature injection, and uses the optimization algorithm to adjust and optimize the parameters of the model.

[0124] Figure 2The NSGA-II optimization method is used to implement the Wensheng graph model fine-tuning description graph. Through population evolution, multiple non-dominated solutions are found on the Pareto front, that is, solutions that perform balanced on multiple objectives. The key features of NSGA-II include: Fast non-dominated sorting: Sort the solutions in the population according to their advantages and disadvantages in the large model fine-tuning multi-objective optimization problem. Crowding distance: It is used to measure the diversity of decision variables to ensure that the solutions are evenly distributed on different objectives and avoid concentration in a local optimal area. Elite retention strategy: Retain the solutions with excellent performance in the previous generation to ensure that the model does not lose valuable information in each evolution.

[0125] Figure 3 It shows the key steps of optimizing model parameters and improving image generation quality by using NSGA-II optimization algorithm in the present invention.

[0126] Figure 4 , Figure 5 and Figure 6 This is a comparison chart of the effects of the present invention considering comprehensive evaluation indicators and other methods; wherein, Figure 4 The NSGA-II optimization algorithm proposed in the present invention is far superior to other comparative optimization algorithms in improving the image generation quality. Figure 5 It is shown that the optimization fine-tuning method proposed by the present invention can effectively ensure the semantic consistency between text and image, and has better effect than other optimization algorithms. Figure 6 It is shown that the content diversity of the present invention is better than other comparison algorithms under different optimization algorithm comparisons. The present invention optimizes and adjusts the basic model of the text graph by using the NSGA-II optimization algorithm to further improve the service quality of generative artificial intelligence.

[0127] Example 2

[0128] This embodiment provides a large model optimization system for a cultural graph based on multi-objective optimization, including:

[0129] The data acquisition module is configured to acquire text data;

[0130] The model building module is configured to build a large model of the Wensheng graph based on the diffusion model;

[0131] The optimization module is configured to construct a multi-objective optimization problem based on the optimization variables, and optimize the parameters of the Wensheng graph model based on the multi-objective optimization problem and using the NSGA-II algorithm;

[0132] The reverse diffusion module is configured to generate the final user demand image by reverse diffusion based on parameter optimization using the Wensheng graph model.

[0133] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a method for optimizing a large model of a Wensheng graph based on multi-objective optimization.

[0134] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the method for optimizing a large model of a Wensheng graph based on multi-objective optimization.

[0135] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for optimizing a large model of a Wensheng graph based on multi-objective optimization, characterized in that: include: Get text data; Construct a large model of the Vincent graph based on the diffusion model; Construct a multi-objective optimization problem based on the optimization variables, and optimize the parameters of the Wensheng graph model based on the multi-objective optimization problem and using the NSGA-II algorithm; Based on parameter optimization, the Wensheng graph model is used for reverse diffusion to generate the final user demand image; The constructing of the large model of the cultural graph based on the diffusion model includes constructing the large model of the cultural graph based on the scalable model architecture of the diffusion model, wherein the reverse generation process of the diffusion model is: The reverse SDE when the condition is generated is: Using the Bayesian formula to derive SDE, it can be expressed as: in, The derivative gradient is 0. Substituting the above formula into the conditions to generate SDE, we can get: in, is the probability of condition y with x as the condition and time t, It is a classification neural network; The multi-objective optimization problem is constructed based on the optimization variables, including establishing an optimization problem for maximizing the image quality of the Wensheng image. The goal is to generate an image with high perceptual quality by adjusting the model input and the generation mechanism, which is expressed as: Among them, image perceptual quality Q, latent variable z, text description t, describes image x; The multi-objective optimization problem constructed based on the optimization variables also includes an optimization problem based on maximum semantic consistency, the goal is to make the text map and the input text description semantically consistent, which is expressed as: Where S(x,t,z) represents the semantic consistency score of the generated image x and text description t, z is a latent variable; The multi-objective optimization problem constructed based on the optimization variables also includes establishing an optimization problem for maximizing content diversity, the goal of which is to generate a set of samples with diverse content, expressed as: Among them, D(X) is the diversity measure of the generated sample set X={x1,x2,…,xn}.

2. The method for optimizing a large model of a Wensheng graph based on multi-objective optimization according to claim 1, characterized in that: The method is based on a multi-objective optimization problem and uses the NSGA-II algorithm to optimize the parameters of the diffusion model, including using the NSGA-II algorithm to optimize the diffusion model parameters with the optimization objectives of maximizing image quality, semantic consistency, and content diversity, and combining the output solution of the optimization algorithm to train the diffusion model to optimize the generation quality of the image.

3. The method for optimizing a large model of a Wensheng graph based on multi-objective optimization according to claim 2 is characterized in that: The method based on parameter optimization and using the Wensheng graph model to perform back diffusion to generate the final user required image includes using a noise predictor to estimate the noise of the image in the latent space and gradually removing the noise to generate a clear image, which is described as: where xt is the image at time step t, θ(xt,t) is the noise estimate of the noise predictor for the current image xt and time step t.

4. A Wensheng graph large model optimization system based on multi-objective optimization, executing the method according to claim 1, characterized in that: include: The data acquisition module is configured to acquire text data; The model building module is configured to build a large model of the Wensheng graph based on the diffusion model; The optimization module is configured to construct a multi-objective optimization problem based on the optimization variables, and optimize the parameters of the Wensheng graph model based on the multi-objective optimization problem and using the NSGA-II algorithm; The reverse diffusion module is configured to generate the final user demand image by reverse diffusion based on parameter optimization using the Wensheng graph model.

5. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing the method according to claim 1 .

6. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the method as claimed in claim 1 .

Images