Method for fine tuning training and optimizing image generation model
By building professional data sets and multiple rounds of fine-tuning training, the image generation model is optimized, and the existing models have insufficient generalization capabilities and inconsistent image generation in the professional field are solved, and high-quality and consistent image generation is achieved.
Patent Information
- Application Number
- CN202410942240.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing fine-tuning training image generation model lacks generalization ability in the professional field, and cannot effectively solve complex application scenarios and segmentation problems, and the generated images have problems such as inconsistent multi-angle and distorted and blurred details.
By constructing a data set of specific majors, using suitable training frameworks and parameters for multiple rounds of fine-tuning training, filtering data that meets the evaluation criteria, gradually optimizing the model, and forming a series of image generation model sets for different segmented tasks.
The model's performance ability and generality for professional segmentation tasks have been improved, and the generated image quality and consistency have been significantly improved, which is suitable for multimodal image generation in complex production processes.
Smart Images

Figure CN119006951A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence, and in particular to a method for fine-tuning, training and optimizing an image generation model. Background Art
[0002] In recent years, with the key breakthroughs in artificial intelligence big model technology and the rapid development trend, the era of generative artificial intelligence AIGC (Artificial Intelligence Generated Content) has gradually arrived. AIGC technology has been widely used in fields that are strongly related to text and image creation (such as scene art, poster design, etc.), and has triggered important changes, fundamentally reshaping the production model of these industries, showing subversive potential.
[0003] Especially in many professions that are highly professional, complex in process, and low in digitization, but at the same time strongly related to images (such as architectural design, landscape design, interior design and other fields in the scheme design stage of the construction industry), AIGC technology has important application prospects and can fundamentally improve the production efficiency of these industries. For example: in the initial concept creation stage of scheme design, conceptual text, sketches, case reference materials, etc. can be input to generate design images for shape selection; in the mid-term in-depth iteration stage of scheme design, a large number of similar schemes can be derived based on the current scheme image with one-click iteration; in the later effect expression stage of scheme design, fast rendering can be performed based on existing model screenshots to generate renderings, etc.
[0004] At present, although there are many image generation models that are fine-tuned and trained based on the open source Stable Diffusion Model in various professions (such as fashion photography, architectural design, character original painting, etc.), which are specifically designed to solve such professional tasks; however, these fine-tuned models generally have the same problems:
[0005] (1) The model has poor generalization ability: a single model cannot fully solve the various complex application scenarios and segmentation problems in the production process of this type of profession. For example, a model fine-tuned and trained for the construction industry cannot achieve image generation for multiple design links in the industry, such as architecture, landscape, and interior design. At the same time, it is unable to generate images of certain buildings in the construction field that are highly professional and have a small amount of data (such as medical buildings, transportation buildings, performance buildings, etc.), and can only generate images of buildings with a large amount of data (such as residential, commercial, and office buildings).
[0006] (2) The model has the underlying problem of inconsistency in multiple angles and distortion and blur of generated image details: Since the stable diffusion model is based on pixels and is guided, predicted, and denoised based on information such as input keywords (prompt), and the basic diffusion model itself is weak in image generation for certain professions. This image generation method is completely different from the underlying logic of traditional image generation through geometric modeling and model rendering in these professions. As a result, the trained model cannot recognize views of the same object from different angles in the same scheme, and will generate inconsistent detailed structures and material colors in different views: for example, the same part of the same building has a glass window material in the image generated on the south facade, and a wooden grille material in the image generated on the north facade. And when these generated images are enlarged, it will be found that their detailed pixels are distorted and blurred: for example, some parts of the straight lines on the building facade will appear curved, and the faces of the background people will appear distorted.
[0007] These problems make it impossible for existing image generation models that have been fine-tuned for various professions to fully solve various problems in the production process of such professions (for example, some models that perform well in building facades and perspective angles perform very poorly from a bird's eye view, and some models that perform well in residential and commercial buildings perform very poorly on public buildings). Professionals are still a long way from fully applying AI technology to actual production to improve efficiency. These problems are particularly evident in multimodal image generation models that have been fine-tuned for professions such as the construction industry and product design. Summary of the invention
[0008] The purpose of this application is to provide a method for fine-tuning training and optimizing an image generation model, which can obtain a series of image generation models for professional segmentation tasks and improve the performance and versatility of the model for professional segmentation tasks.
[0009] In a first aspect, the present application provides a method for fine-tuning training and optimizing an image generation model. The method comprises:
[0010] S1: Construct the first dataset for a specific major;
[0011] S2: using the first data set, taking the first image generation model as the basic model, adopting the first training framework, setting appropriate parameters to perform fine-tuning training on the first image generation model, and obtaining a second image generation model;
[0012] The first training framework is a training framework suitable for the first data set and the professional task requirements;
[0013] S3: Filter out the data in the first data set that meets the professional evaluation standards and quality requirements to obtain a second data set;
[0014] S4: using the second data set, taking the second image generation model as the basic model, adopting the first training framework, setting appropriate parameters to perform fine-tuning training on the second image generation model, and obtaining a third image generation model;
[0015] Through the fine-tuning training in this step, the model's performance in comprehensive tasks in all aspects of this profession can be enhanced.
[0016] S5: Segment the first data set according to different segmented tasks of the profession to obtain multiple third data sets;
[0017] Each third data set is a sub-dataset of the first data set corresponding to a subdivided task of the profession;
[0018] S6: using the plurality of third data sets respectively, taking the third image generation model as a basic model, adopting the first training framework, and adjusting appropriate parameters according to the characteristics of the corresponding third data sets respectively, performing fine-tuning training on the third image generation model, and obtaining a plurality of fourth image generation models;
[0019] Each fourth image generation model corresponds to a subdivision task of the profession; the plurality of fourth image generation models are a series of image generation models for different subdivision tasks of the profession;
[0020] S7: Filter out the data that meet the corresponding evaluation criteria and quality requirements of the professional subdivision tasks from the plurality of third data sets, and obtain a plurality of fourth data sets accordingly;
[0021] S8: Using the plurality of fourth data sets respectively, taking the fourth image generation model as the basic model, adopting the first training framework, setting appropriate parameters to perform fine-tuning training on the fourth image generation model, and obtaining a plurality of fifth image generation models respectively.
[0022] Through the fine-tuning training in this step, the model's performance for the corresponding professional sub-task can be enhanced.
[0023] In some possible implementations, in S2, a stable diffusion model is selected as the first image generation model, and Dreambooth is selected as the first training framework.
[0024] In some possible implementations, during the fine-tuning training process, the model training progress is monitored and evaluated through sample images, loss value curves, and learning rate curves to determine whether there is a need to terminate the training early to adjust the model structure and training parameters.
[0025] In some possible implementations, during each round of fine-tuning training, the process model of each preset number of training steps and the final model of all training cycles are saved; each process model, the final model and the basic model are deployed to the environment, the same parameters are input, and the image generation effects of each process model, the final model and the basic model for the professional task are compared, and the model with the best overall performance is selected as the final model of this round of fine-tuning training.
[0026] In some possible implementations, in S4, multiple rounds of fine-tuning training are performed, and in each round, the model optimized by the previous round of training is used as the basic model for repeated iterations until the optimized model reaches the first preset performance in various subdivided tasks of the profession.
[0027] In some possible implementations, in S8, multiple rounds of fine-tuning training are performed, and in each round, the model optimized by the previous round of training is used as the basic model for repeated iterations until the optimized model reaches a second preset performance on the professional specific category segmentation task.
[0028] In some possible implementations, each round of fine-tuning training process adjusts the structure and training parameters of the model according to the characteristics and amount of the data set, with reference to previous training results and evaluation indicators.
[0029] In a second aspect, the present application provides an electronic device, including: a memory and a processor;
[0030] Memory for storing computer programs;
[0031] A processor is used to call a computer program to perform the above method of fine-tuning training and optimizing the image generation model.
[0032] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program runs on an electronic device, the electronic device implements the above-mentioned method of fine-tuning training and optimizing the image generation model.
[0033] In a fourth aspect, the present application provides a computer program product, including a computer program. When the computer program runs on an electronic device, the electronic device implements the above-mentioned method of fine-tuning training and optimizing the image generation model.
[0034] The specific implementation methods of the second to fourth aspects of the present application can refer to the implementation methods of the first aspect, which will not be repeated here.
[0035] Beneficial effects:
[0036] (1) Improve data usability
[0037] The structured classification and phased training of professional data sets reduce the duplication of data, improve the usability of data, and strengthen the management of professional data. The fine-tuning training ideas proposed in this application have important reference value for how to better apply AI technology to traditional industries with complex production processes, huge data volumes, and low levels of informatization.
[0038] (2) Improve the performance and versatility of the model for specialized tasks and enhance the implementation of AI technology in traditional industries
[0039] The continuous use of data-driven and optimized model training mechanism in stages enables the final optimized series of professional subdivision task image generation model sets to perform well in dealing with various problems in complex production processes. Different sub-models can accurately understand and analyze professional requirements, quickly generate high-quality initial solutions, and quickly make corresponding adjustments when faced with changes and adjustments in constraints to maintain the consistency and accuracy of the output images. In addition, the self-learning and optimization capabilities of the model enable it to continuously accumulate experience in practice and improve image quality. The fine-tuning training scheme proposed in this application has important reference value for how to better apply AI technology to traditional industries to solve practical engineering problems, and is suitable for fine-tuning training and optimization of professional multimodal image generation models. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solution of the embodiment of the present application, the following is a brief introduction to the drawings required for use in the embodiment. It should be understood that the following drawings Figure 1 To Attachment Figure 3 Only certain embodiments of the present application are shown, and therefore should not be regarded as limiting the scope of protection of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.
[0041] Figure 1 This is a flow chart of a method in one embodiment of the present application;
[0042] Figure 2 A flow chart of training and optimizing a professional basic image generation model in one embodiment of the present application;
[0043] Figure 3 A flowchart of training and optimizing a set of image generation models for a series of professional segmentation tasks in one embodiment of the present application; DETAILED DESCRIPTION
[0044] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0045] In order to make the embodiments of the present application clearer, the meanings of the relevant terms are first explained below.
[0046] Professional images: All image materials with the knowledge and experience required for certain fields and professions that are strongly related to images (such as architectural design, interior design, landscape design, etc.) are generated during the production process. For example, in the production process of architectural design, all image materials (including but not limited to case reference drawings, hand-drawn drawings, analysis drawings, electronic model drawings, renderings (effects) drawings, etc.) that express architectural design-related factors (such as the concept source of the plan, volume relationship, perspective angle, material color, etc.) are generated.
[0047] Multimodal image generation model: A type of AI model that can generate images associated with input content based on input text, images, data and other parameters, such as the Stable Diffusion model and the Generative Adversarial Network (GAN), which can be fine-tuned and trained as a basic model.
[0048] Professional multimodal image generation model: By collecting a certain type of professional image dataset (such as an image dataset for architectural design) and feature annotations, a general training framework is used to fine-tune the basic multimodal image generation model to obtain a professional multimodal image generation model, and enhance the performance of the basic model when processing professional tasks. For stable diffusion models, the commonly used fine-tuning training frameworks include Dreambooth (mainly used to strengthen the overall basic model) and LoRA (low-rank training framework, mainly used to train small models).
[0049] Multi-stage fine-tuning training: Use the same training framework again, and fine-tune the professional image dataset of this type, and fine-tune the fine-tuned model again to further optimize the model's performance in processing this professional task. This optimization process can be repeated multiple times.
[0050] Serial model set: According to different tasks in a certain type of professional production process, the professional image dataset is subdivided into different sub-datasets (for example, the architectural industry scheme design dataset can be subdivided into architectural design dataset, landscape design dataset, and interior design dataset). Each sub-dataset uses the same training framework and fine-tunes the same model as the base model to generate a series of sub-models derived from the same model that have strong performance capabilities for different sub-tasks of the profession. All sub-models together constitute a model set.
[0051] This application takes into account the production design process of professions that are strongly related to images, which is a complex process that requires comprehensive consideration of various factors, with many subdivisions and a step-by-step approach. If a single image generation model is formed by simply mixing a small or large number of image data sets of a certain profession and fine-tuning the training for the profession, since these mixed data sets are not structurally divided and phased according to the various subdivisions of the profession, the single model trained once cannot rely solely on input text, images and other conditions to determine how to better express this aspect when processing the professional subdivision tasks.
[0052] Therefore, an embodiment of the present application proposes a method for fine-tuning training and optimizing an image generation model: by structurally subdividing a large number of datasets in a profession (such as image datasets used for fine-tuning training of multimodal image generation models), dividing them into a first dataset, a second dataset, multiple third datasets, and multiple fourth datasets, and performing phased, multi-round fine-tuning training on the different sub-datasets after subdivision, continuously optimizing and refining the image generation model's performance capabilities for the professional comprehensive tasks and different subdivided tasks, and ultimately obtaining an optimized set of image generation models for a series of professional subdivided tasks, so that each sub-model in the image generation model set can better solve the various complex subdivision problems encountered in the actual production of the corresponding profession.
[0053] The embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0054] In a first aspect, the present application provides a method for fine-tuning training and optimizing an image generation model. Figure 1 As shown, the method comprises the following steps:
[0055] S1: Construct the first data set (professional basic data set) for a specific major;
[0056] In some embodiments, step S1 includes: for a specific profession, collecting a large amount of image data related to the profession and preprocessing the data, generating corresponding feature annotations, and obtaining a basic data set belonging to the profession as the first data set.
[0057] S2: using the first data set, taking the first image generation model (basic image generation model) as the basic model, adopting the first training framework, setting appropriate parameters to fine-tune the first image generation model, and obtaining the second image generation model (professional basic image generation model); the first training framework is a training framework suitable for the first data set and the professional task requirements;
[0058] S3: Filter out the data in the first data set that meet the professional evaluation standards and quality requirements to obtain the second data set (fine screening professional basic data set);
[0059] According to the professional evaluation standard and the image quality evaluation standard, the image data in the first data set is screened to select images that meet the professional evaluation standard and have good quality. Among them, the professional evaluation standard refers to whether the main content of the image meets the requirements of the profession, such as whether the car in the car image is a car that meets the requirements, and whether the main building in the building image is a building that meets the requirements; quality refers to the basic quality of the image, such as clarity, color saturation, etc.
[0060] S4: Using the second data set, taking the second image generation model as the basic model, adopting the first training framework, setting appropriate parameters to fine-tune the second image generation model, and enhancing its performance for various comprehensive tasks of the profession (such as generating better image surface effects, and product content that is more in line with professional evaluation standards), and obtaining the third image generation model (optimized professional basic image generation model);
[0061] In some embodiments, the optimization process in step S4 can be repeated for multiple rounds, and each round uses the model optimized in the previous round of training as the basic model for repeated iterations until the optimized model reaches the first preset performance in various subdivided tasks of the profession, that is, exhibits good overall performance.
[0062] The following combination Figure 2 , the key steps and components in training and optimizing the professional basic image generation model (S1-S4) in the embodiment of the present application are described in detail:
[0063] Construct the first and second datasets: For example, take the dataset screening and quantity standards for model training in the design phase of the construction industry as an example, and other professions can also refer to it. In this example, the dataset screening standards include: resolution, the shortest side of the image should be at least 1024 pixels to ensure image quality and clarity; professional evaluation standards, the main content of the dataset should meet the evaluation standards of the professional experience, such as for the construction industry, the main building of the image should have good visual beauty, rendering effect or shooting quality (such as high-quality architectural renderings, or real-life photos of buildings with clean and beautiful pictures); main content, the main content of the image should be clear and complete, with a moderate distance to show the overall scene, and images that are too local, too far away or unclear in theme should be avoided. After the first dataset is obtained by screening according to the above standards, the scope of the standards is further narrowed, the requirements are improved, and the second dataset is screened out from the first dataset. Through screening, the final dataset size determined in this example is as follows: the first dataset contains more than 100,000 screened image-text pairs, covering a variety of product types and scenes. The second dataset contains more than 10,000 screened high-quality image-text pairs.
[0064] Set the training framework and training parameters: select the basic model for training and the corresponding format, and create the basic model from safetensors or ckpt and other formats into a format that can be used for secondary training of the training framework. The existing performance of the basic model must be suitable to a certain extent for various tasks in the profession. In some embodiments, an open source stable diffusion model (such as the stable-diffusion-xl-base-1.0 model) is selected as the base model. Selecting an open source original model usually has a more balanced performance in various professions, which can ensure the versatility and comprehensiveness of the model; if you choose an image generation model that has been fine-tuned and trained for certain professions (this type of model usually has a professional preference), if you choose an appropriate one based on task requirements and authorization, it can reduce the difficulty of training to a certain extent.
[0065] The training framework includes the scripts used to train the model, the required configuration parameters, etc. In the script of the training framework, some parameters are configured and fixed, and some need to be manually adjusted, such as learning rate, training rounds, etc., which need to be manually controlled.
[0066] In some embodiments, Dreambooth is selected as the fine-tuning training framework of the stable diffusion model, that is, the first training framework. For example, the specific training steps and parameters for the field of the scheme design stage of the construction industry (other professions can refer to it) are as follows:
[0067] Dataset settings: Dataset settings are divided into training target dataset settings and regularization dataset settings. The training target dataset is the main target to be trained, and its settings are as follows: (1) Select the storage path of the image dataset (the first dataset or the second dataset) and the annotation document corresponding to the image dataset; (2) Set identifiers, including instance identifiers and category identifiers. The instance identifier will use a special character as the binding of the training set and be added to the understanding of the model through training. The category identifier is the category to which the training target image concept belongs. For example, taking the method of setting identifiers during model training in the design stage of the construction industry as an example, other professions can also refer to it: "building" is selected as the category identifier to avoid language drift and overfitting in the model after training, which affects the understanding of other things in the model. The regularization dataset uses the original basic model to automatically generate images for the category identifier "building" in a ratio of 1:2 to the number of images in the training target dataset to better control the model prior loss.
[0068] Training parameter setting: The setting of training parameters mainly involves determining the optimal learning rate (including the selection of initial learning rate, optimizer and learning rate scheduler) and other parameters (batch size, number of training steps, etc.).
[0069] For example, the method of determining the optimal learning rate during model training in the design phase of the construction industry is taken as an example, and other majors can also refer to it: select the adaptive optimizer Adafactor, set the initial learning rate to 1, select the corresponding Adafactor scheduler for the learning rate scheduler, set the learning rate warm-up steps to 0, start training and observe the learning rate curve during training, select the value after the curve stabilizes as the optimal learning rate and terminate the training, reselect the optimizer AdamW, set the learning rate to the determined optimal learning rate, select liner for the learning rate scheduler, set the learning rate warm-up steps to 50, and continue training until the end of training. The remaining training parameter settings include: enabling EMA to alleviate video memory usage, using bf16 mixed precision and turning on full mixed precision, enabling xformers to optimize video memory usage, setting CLIP skip layers to 2, setting the resolution of image data processing to 1024*1024, and enabling data bucketing for the corresponding resolution, setting the batch size to 16 according to the video memory and data volume, setting the epoch (training cycle) to 100, saving samples at a frequency of every 1% of the training steps, and saving process models at a frequency of every 10% of the training steps.
[0070] The training process based on the above parameter settings is as follows: After loading the original weights of the basic model, the training framework loads the image dataset, inputs the image into the cached latent space and automatically buckets it, and classifies images with different aspect ratios into corresponding groups. Load the parameters set above and calculate the total number of steps required for training. Input the data into the model batch by batch according to the parameters, and perform forward propagation, loss calculation, back propagation, and parameter update. Repeat the process of forward propagation, loss calculation, back propagation, and parameter update, and output intermediate samples and intermediate models according to the settings until the specified stopping condition is reached (such as reaching the maximum number of training rounds, loss value convergence, etc.). Each iteration will update the model parameters in a more optimal direction, so that the model gradually converges to the optimal solution.
[0071] Training monitoring: In some embodiments, monitoring is performed through sample images, loss value curves, and learning rate curves to evaluate the training progress of the model and identify potential problems. The specific methods are as follows: Observe the sample images. At the end of each 1% of the training steps, use the current fine-tuned model to generate and save images according to the set sample parameters. Observe the prediction results of the samples, pay attention to whether there are obvious errors or deviations, and pay attention to the changing trends of images under the same seed. Monitor the loss function. The training framework will continuously record the loss function according to the settings, draw a loss value curve, observe the changes in the loss curve, and pay attention to whether the loss value gradually decreases and stabilizes, or whether there are drastic fluctuations or sudden increases. Monitor the learning rate curve: The training process continuously records and draws a learning rate curve, observes the changes in the learning rate curve, and pays attention to whether the learning rate is adjusted as expected, and whether there is a need to terminate the training early to adjust the learning rate.
[0072] Model comparison: After training is completed, the process model of every 10% training steps and the final model of all training cycles will be saved as set. All models are deployed to the environment, the same inference parameters are input, and the image generation effects of each process model, the final model and the original basic model for the professional task are compared. The model with the best overall performance is selected as the final trained model.
[0073] The embodiment of the present application uses the Dreambooth training framework to generate a complete first image generation model, which can be independently deployed for reasoning. Its versatility and functionality in actual production are far higher than the existing small model mounted on the first image generation model for reasoning.
[0074] Repeat the optimization phase:
[0075] After training the basic model for the professional task, the trained model is used as the basic model, the second data set is used, and the same training framework is used to readjust the appropriate parameters and train again to further optimize the performance of the trained model in processing the professional task. This stage can be repeated N times for continuous optimization. The training parameters that can be adjusted and optimized each time include: (1) structural adjustment of the model: the training rate of the convolutional network U-net and the training rate of the text encoder, the skipping layer of the CLIP model, the training accuracy of the model, the noise offset, the weight attenuation degree, etc.; (2) other parameters: the training batch size and epoch are scaled according to the corresponding proportion of the data set, and the learning rate is determined in the same way as the training framework in the first stage.
[0076] The training time for each round of optimization process will vary depending on factors such as model complexity, data set size, hardware configuration, etc. According to the above parameter settings, the data volume is about 1w, using A100 (80G VRAM), AdamW optimizer, and a batch size of 8, a round of optimization training usually takes about 300h. Generally speaking, the fine-tuning training in the design stage of the construction industry targeted by this example can obtain a third image generation model that shows good comprehensive performance in various subdivided tasks of this profession through 2 to 3 stages of repeated optimization;
[0077] S5: Segment the first data set according to different subdivision tasks of the profession to obtain multiple third data sets (professional subdivision sub-data sets); each third data set is a sub-data set of the first data set corresponding to a subdivision task of the profession;
[0078] Among them, for the different sub-tasks of this major, you can choose all the sub-tasks of this major, or you can choose part of the sub-tasks under this major according to your needs.
[0079] S6: using the third image generation model as a basic model using the first training framework, respectively using the plurality of third data sets, and adjusting appropriate parameters according to the characteristics of the corresponding third data sets, and performing fine-tuning training on the third image generation model to obtain a plurality of fourth image generation models (professional subdivision task image generation sub-models); each fourth image generation model corresponds to a subdivision task of the profession; and the plurality of fourth image generation models are a series of image generation models for different subdivision tasks of the profession;
[0080] S7: Filter out the data that meet the corresponding professional subdivision task evaluation criteria and quality requirements from the multiple third data sets, and obtain multiple fourth data sets (finely screened professional subdivision sub-data sets) accordingly;
[0081] Among them, the professional subdivision task evaluation criteria refers to whether the main content of the image meets the requirements of the professional subdivision task. Through this step, images that meet the professional subdivision task evaluation criteria and have good quality are screened out.
[0082] The fourth data set is a data set obtained by finely screening the data in the corresponding third data set for the corresponding professional segmentation task.
[0083] S8: Using multiple fourth data sets respectively, taking the fourth image generation model as the basic model, adopting the first training framework, setting appropriate parameters to fine-tune the fourth image generation model, and enhancing its performance for corresponding professional subdivision tasks (such as the model for the subdivision task of green buildings in the architectural design industry can more accurately generate building images that meet the professional evaluation criteria for green buildings), and respectively obtaining multiple fifth image generation models (optimized professional subdivision task image generation sub-models).
[0084] This optimization process can be repeated for multiple rounds, and the optimized model in each round of training is used as the basic model for the next round of training, until the fifth image generation model for the professional specific category segmentation task reaches the second preset performance, that is, it shows excellent performance.
[0085] All fifth image generation models together form an optimized set of image generation models for a series of professional segmentation tasks.
[0086] The following combination Figure 3 , the key steps and components in training and optimizing the image generation model set (S5-S8) for a series of professional subdivision tasks in the embodiment of the present application are described in detail:
[0087] Construct the third and fourth data sets: Exemplarily, take the professional data set segmentation and logic for model training in the scheme design stage of the construction industry as an example, and other professions can also refer to it. The subdivided sub-dataset contains the following categories: According to the different professions in the scheme design stage of the construction industry, it is divided into four third data sets: architectural design, interior design, landscape design, and urban design. Each sub-dataset contains processed images and corresponding annotation documents. The subdivided sub-datasets have carried out in-depth mining and more detailed annotations for each subdivision of the profession. According to the same standard screening as step S3, the standard range is further narrowed, the requirements are improved, and the fourth data set is screened in the subdivided sub-dataset. Finally, the data volume of each third data set is about 30,000 to 70,000, and the data volume of the fourth data set is about 3,000 to 5,000.
[0088] Set the training framework and training parameters:
[0089] In some embodiments, during the training and optimization of the image generation model set for a series of professional subdivision tasks, the same Dreambooth fine-tuning training framework as in S1 to S4 is still used as the stable diffusion model. The specific training steps and parameters of this stage are as follows:
[0090] Create the same base model for all submodels:
[0091] For all third data sets, the third image generation model trained in the previous stage is selected as the base model for fine-tuning training. The corresponding model category and safetensors format are selected to create a model that can be trained by the training framework.
[0092] Dataset settings:
[0093] The training target data set is set to the corresponding third data set in turn. This application takes the method of setting identification words when training the model in the design stage of the construction industry as an example. Other professions can also refer to it: for each sub-dataset, "Architecture Design", "Landscape Design", "Interior Design", and "Urban Design" are selected as their category prompt words when training their sub-models. The regularized data set of each sub-dataset still uses the category prompt word "building" in S1 and S2, and automatically generates images in a ratio of 1:2 to the number of images in the training target data set, so as to better control the model prior loss.
[0094] Training parameter settings:
[0095] The determination of the optimal learning rate (including the selection of the initial learning rate, optimizer and learning rate scheduler) is consistent with the determination method when training the professional basic model in S1 to S4. The remaining parameters (batch size, number of training steps, etc.) can be adjusted in appropriate proportions according to the amount of data in the third data set. This application sets the batch size to 4 and the epoch to 100 when training each sub-model at this stage. Based on the above settings, training is performed, and the process of forward propagation, loss calculation, back propagation and parameter update is repeated until the specified stop condition is reached (such as reaching the maximum number of training rounds, loss value convergence, etc.).
[0096] Training monitoring:
[0097] In this stage, the same training monitoring method as in S1 to S4 is adopted.
[0098] Sub-model comparison:
[0099] After the training is completed, the process sub-models of every 10% training steps and the final sub-model after all cycles are saved as set. All sub-models are deployed to the environment, the same inference parameters are input, and the image generation effects of each process model, the final model and the original basic model for this professional sub-task are compared. The sub-model with the best overall performance is selected as the final trained model.
[0100] Repeat the optimization phase:
[0101] Based on the fourth image generation model (professional segmentation task image generation model) obtained in S6, each segment field can adjust the model structure and training parameters for optimization training according to the characteristics and data volume of the fine-screened dataset, and refer to previous training results and evaluation indicators. During multiple rounds of repeated optimization, there are slight differences in the parameter settings of each round, which are determined based on the model effects and datasets of continuous iterations. Optionally, improvements in training parameters may include: (1) Structural adjustment of the model: the training rate of the convolutional network U-net and the training rate of the text encoder, skipping layers of the CLIP model, training accuracy of the model, noise offset, weight attenuation, etc.; (2) Other parameters: training batch size and epoch are scaled according to the corresponding proportions of the dataset, etc.
[0102] According to the above parameter settings, the data volume is about 5K, using A6000 (48G VRAM), AdamW optimizer, and batch size 8, it usually takes about 200 hours to perform one round of optimization training. Generally speaking, the fine-tuning training of the subdivision field of the construction industry scheme design stage targeted by this application can obtain an optimized professional subdivision task image generation sub-model with excellent performance in this professional subdivision task through 2 to 3 stages of repeated optimization.
[0103] The optimized image generation model set for a series of professional segmentation tasks (final results):
[0104] Among all the subdivided tasks of this profession, the optimized professional subdivided task image generation sub-models obtained after multiple rounds of optimization together constitute the optimized serial professional subdivided task image generation model set. This serial model set, based on the inheritance of the third image generation model, further improves the performance in each subdivided task, and has a strong adaptability to specific professional subdivided tasks. At the same time, after multiple rounds of optimization training, it can better understand and accurately reflect the characteristics and needs of images in the subdivided field.
[0105] The phased training and optimization method of the series of sub-models proposed in this application can solve a series of sub-problems encountered in the use of AI technology for actual image production in various professions that are strongly related to images: including hand-drawn conversion diagrams, conceptual image diagrams, reference diagrams in the early stage of the plan, derivative and in-depth diagrams in the process of scheme design, and renderings in the later stage of scheme design, etc., to realize the application of image generation models in various image-related professions.
[0106] An embodiment of the present application also provides an electronic device, including: a memory and a processor;
[0107] The memory is used to store computer programs;
[0108] The processor is used to call the computer program to execute the method of fine-tuning training and optimizing the image generation model as described above.
[0109] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed on an electronic device, the electronic device implements the method for fine-tuning training and optimizing the image generation model as described above.
[0110] An embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed on an electronic device, the electronic device implements the method for fine-tuning training and optimizing the image generation model as described above.
[0111] The specific implementation methods of the device, electronic device, computer-readable storage medium, and computer program product provided in the embodiments of the present application can refer to the specific embodiments of the above-mentioned method for fine-tuning training and optimizing the image generation model, which will not be repeated here.
[0112] Obviously, those skilled in the art should understand that the above-mentioned units or steps of the present application can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software.
[0113] The above description of the embodiments of the present application is only a partial embodiment of the present application, which is used to enable professionals in the field to implement or use the content of the present application, and is not used to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A method for fine-tuning, training and optimizing an image generation model, characterized in that: include: S1: Construct the first dataset for a specific major; S2: using the first data set, taking the first image generation model as a basic model, adopting a first training framework, setting appropriate parameters to perform fine-tuning training on the first image generation model, and obtaining a second image generation model; S3: Filter out the data in the first data set that meets the professional evaluation standards and quality requirements to obtain a second data set; S4: using the second data set, taking the second image generation model as a basic model, adopting the first training framework, setting appropriate parameters to perform fine-tuning training on the second image generation model, and obtaining a third image generation model; S5: Segment the first data set according to different segmented tasks of the profession to obtain multiple third data sets; Each third data set is a sub-dataset of the first data set corresponding to a subdivided task of the profession; S6: respectively using the multiple third data sets, taking the third image generation model as a basic model, adopting the first training framework, and adjusting appropriate parameters according to the characteristics of the corresponding third data sets, fine-tuning the third image generation model to obtain multiple fourth image generation models; S7: Filter out the data that meet the corresponding professional subdivision task evaluation criteria and quality requirements from the multiple third data sets respectively, and obtain multiple fourth data sets accordingly; S8: Using the plurality of fourth data sets respectively, taking the fourth image generation model as the basic model, adopting the first training framework, setting appropriate parameters to perform fine-tuning training on the fourth image generation model, and obtaining a plurality of fifth image generation models respectively.
2. The method according to claim 1, characterized in that In S2, a stable diffusion model is selected as the first image generation model, and Dreambooth is selected as the first training framework.
3. The method according to claim 1, characterized in that During the fine-tuning training process, the training progress of the model is monitored and evaluated through sample images, loss value curves, and learning rate curves to determine whether there is a need to terminate the training early to adjust the structure and training parameters of the model.
4. The method according to claim 1, characterized in that: During each round of fine-tuning training, the process model of each preset training step and the final model of all training cycles are saved; each process model, the final model and the basic model are deployed to the environment, the same parameters are input, and the image generation effects of each process model, the final model and the basic model for the professional task are compared, and the model with the best overall performance is selected as the final model of this round of fine-tuning training.
5. The method according to claim 1, characterized in that In S4, multiple rounds of fine-tuning training are performed, and in each round, the model optimized by the previous round of training is used as the basic model for repeated iterations until the optimized model reaches the first preset performance in various subdivided tasks of the profession.
6. The method according to claim 1, characterized in that In S8, multiple rounds of fine-tuning training are performed, and in each round, the model optimized by the previous round of training is used as the basic model for repeated iterations until the optimized model reaches the second preset performance on the professional specific category segmentation task.
7. The method according to claim 5 or 6, characterized in that: In each round of fine-tuning training, the model structure and training parameters are adjusted according to the characteristics and amount of the dataset, with reference to previous training results and evaluation indicators.
8. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor is configured to call the computer program to execute the method according to any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on an electronic device, the electronic device implements the method according to any one of claims 1 to 7.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed on an electronic device, the electronic device implements the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Method and device for establishing intelligent model in communication professional subdivision scene
CN117520840A
Multi-lingual natural language generation
US20240127008A1
System and methods for tuning ai-generated images
US20240161258A1
