A small-sample fine-tuning method based on regularization constraints to prevent catastrophic forgetting.

Abstract

This invention provides a few-sample fine-tuning method based on regularization constraints to prevent catastrophic forgetting. A preliminary customized model is obtained by training a basic Stable Diffusion model using standard LoRA on small sample data. During continued training, small sample data and auxiliary data are fused. A regularization loss term to prevent catastrophic forgetting is introduced into the loss function. This regularization loss term constrains the consistency between the current model output and the previous model output, reducing the destruction of learned features during new sample learning, thereby improving the model's generalization ability and stability under the target style. This method balances rapid model adaptation to new samples with the preservation of existing capabilities, avoiding problems such as style drift and loss of detail.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a small-sample fine-tuning method based on regularization constraints to prevent catastrophic forgetting. Background Technology

[0002] Stable Diffusion and other diffusion models have been widely used in text-to-image tasks. LoRA (Low-Rank Adaptation) is a mainstream method for fine-tuning with few samples, enabling efficient model customization with only a dozen or so samples. However, in some complex application scenarios (such as game characters or custom anime-style IPs), the model may struggle with style consistency and detail preservation when learning the target task, resulting in unstable or inconsistent generated results.

[0003] Based on this, the present invention proposes a small-sample fine-tuning method based on regularization constraints to prevent catastrophic forgetting. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a small-sample fine-tuning method based on regularization constraints to prevent catastrophic forgetting.

[0005] The technical solution adopted in this invention is as follows:

[0006] A small-sample fine-tuning method based on regularization constraints to prevent catastrophic forgetting includes the following steps:

[0007] S1: Obtain the basic Stable Diffusion model and the few-shot training set; wherein, the few-shot training set contains N sets of text-image pairs;

[0008] S2: Initialize the low-rank adapter and insert it into the cross-attention layer of the base Stable Diffusion model; freeze the original weights of the base Stable Diffusion model; and train the low-rank adapter based on the small sample training set to obtain a primary customized model by minimizing the image reconstruction loss function.

[0009] S3: Merge the small sample training set and the auxiliary dataset into a mixed training dataset, wherein the auxiliary dataset is an auxiliary dataset related to the image generation task;

[0010] S4: Load the primary customized model as a reference model, and create a complete copy of the primary customized model as the current fine-tuning model;

[0011] S5: Perform dual-path forward computation on each sample in the mixed training dataset, extract reference output features through the reference model, and extract current output features through the current fine-tuning model;

[0012] S6: Calculate the catastrophic forgetting regularization loss term based on the distribution difference between the reference output features and the current output features;

[0013] S7: Calculate the image reconstruction loss function based on the hybrid training dataset and the current fine-tuning model; combine the calculated image reconstruction loss function and the catastrophic forgetting regularization loss term, and construct a joint loss function by adding them according to the preset weight coefficients of the catastrophic forgetting regularization loss term; update the low-rank adapter parameters in the current fine-tuning model through backpropagation;

[0014] S8: Repeat S5 to S7 until the set termination condition is met, and then stop and output the current fine-tuned model as the final optimized model.

[0015] In S2, the low-rank adapter is a LoRA adapter, which consists of a pair of low-rank matrices, and the rank r of the low-rank matrix satisfies: 4≤r≤64; its parameter update only applies to the weight matrix of the cross-attention layer of the basic Stable Diffusion model.

[0016] In step S2, the freezing operation of the original weights of the basic Stable Diffusion model includes: setting the weights of all linear layers of the basic Stable Diffusion model to an untrainable state, and retaining only the parameters of the low-rank adapter as trainable.

[0017] In step S3, the construction of the auxiliary dataset includes: obtaining auxiliary data from data sources related to the image generation task, including but not limited to public image libraries, historical generated image data, and pre-trained model feature data; and performing deduplication and image quality filtering on the obtained auxiliary data to obtain the auxiliary dataset.

[0018] In step S5, both the reference output feature and the current output feature are image latent space feature vectors, which are extracted through the decoder output layer of the reference model and the current fine-tuning model.

[0019] In step S6, the distribution difference between the reference output feature and the current output feature is measured by KL divergence.

[0020] In step S7, the weight coefficient of the regularization loss term for preventing catastrophic forgetting is set to Q1 in the early stage of training and Q2 in the later stage of training, wherein Q1>Q2.

[0021] In step S7, the calculation of the image reconstruction loss function includes: inputting text prompts from the mixed training dataset into the current fine-tuning model to obtain the corresponding model-generated image; calculating the image reconstruction loss function based on the difference between the model-generated image and the real image in the mixed training dataset; wherein the image reconstruction loss function satisfies the following conditions:

[0022] When comparing the model-generated image with the real image in pixel space, the mean squared error loss function of the RGB channels is used;

[0023] When comparing the model-generated image with the real image in the latent space, the L1 norm loss function of the latent variable features is used.

[0024] In S7, the updated low-rank adapter parameters include: weight parameters of the low-rank matrix pairs in the cross-attention layer; bias term parameters associated with the low-rank matrix pairs; and the original weight gradients of the underlying Stable Diffusion model are masked during the update process.

[0025] In S8, the termination condition is set as follows: S5 to S7 are repeated until the preset maximum number of training rounds is reached or the decrease in the value of the joint loss function for K consecutive rounds is less than a preset threshold.

[0026] The beneficial effects of this invention are:

[0027] This invention uses small sample data to train a basic Stable Diffusion model using standard LoRA to obtain a preliminary customized model. During further training, small sample data and auxiliary data are integrated. A regularization loss term to prevent catastrophic forgetting is introduced into the loss function. This regularization loss term constrains the consistency between the current model output and the previous model output, reducing the destruction of learned features during new sample learning, thereby improving the model's generalization ability and stability under the target style. It balances the model's rapid adaptation to new samples with the preservation of existing capabilities, avoiding problems such as style drift and loss of detail. Attached Figure Description

[0028] Figure 1 This is an overall flowchart of the small-sample fine-tuning method for preventing catastrophic forgetting based on regularization constraints, according to an embodiment of the present invention.

[0029] Figure 2 This is a flowchart illustrating the initialization of the LoRA adapter and training of a primary custom model according to an embodiment of the present invention;

[0030] Figure 3 This is a flowchart of a dual-path forward calculation according to an embodiment of the present invention;

[0031] Figure 4 This is a flowchart illustrating a method for calculating the regularization loss term to prevent catastrophic forgetting according to an embodiment of the present invention.

[0032] Figure 5 This is a flowchart illustrating the construction and parameter update mechanism of the joint loss function according to an embodiment of the present invention.

[0033] Figure 6 This is a flowchart illustrating the logic for determining the termination condition according to an embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] like Figure 6 As shown, S8: Repeat S5 to S7 until the set termination condition is met, and then stop, and output the final updated current fine-tuned model as the final optimized model.

[0036] S1: Obtain the base Stable Diffusion model and the few-shot training set. The base Stable Diffusion model is a pre-trained image generation model that serves as the foundation for subsequent fine-tuning. The few-shot training set contains N text-image pairs, denoted as... .

[0037] in, Indicates the first The text prompts for the group describe the content of the image to be generated in natural language, for example, "an eagle flying in the sky". These text prompts provide semantic guidance for the model to generate the image. This represents the real image corresponding to the text prompt, which serves as a supervisory signal to guide the model's training process, ensuring that the model can learn the correct mapping relationship from text to image.

[0038] When acquiring a small training set, data collection can be tailored to the specific application scenario. For example, if the goal is to generate images of anime characters in a specific style, corresponding anime character images and their descriptive text can be collected from anime databases, relevant art community websites, and other sources. Simultaneously, to ensure data quality and validity, the collected data needs to be filtered and preprocessed. Filtering primarily involves checking the image clarity, completeness, and consistency between the text description and the image content, eliminating blurry, damaged, or inaccurate data samples. Preprocessing operations may include image size normalization, adjusting all images to the same resolution for model input, and text segmentation and vectorization, converting natural language into a numerical form that the model can process.

[0039] like Figure 2 As shown, S2: Initialize the low-rank adapter and insert it into the cross-attention layer of the base Stable Diffusion model; freeze the original weights of the base Stable Diffusion model; and train the low-rank adapter based on a small training set. By minimizing the image reconstruction loss function, a primary customized model is obtained.

[0040] The preliminary customization model obtained in this step can reflect the characteristics of the small sample training set to a certain extent. For example, in the game character customization scenario, if the small sample training set contains unique character designs, clothing styles, and action designs from a popular game, the preliminary customization model can initially generate new character images with similar clothing textures, iconic weapons, and action postures. In the field of ACG (Anime, Comics, and Games) IP customization, when the small sample training set consists of classic character illustrations from a well-known ACG IP, the preliminary customization model can produce new character concept art with similar art style, color schemes, and character design characteristics. However, since only a small number of samples were used for training, the model may encounter problems such as disproportionate character shapes, blurred clothing details, or jittery lines and abnormal color saturation in the ACG character art style during the generation process. Subsequent steps further optimize these issues by introducing auxiliary datasets and using catastrophic forgetting regularization, enabling the model to better adapt to specific task requirements while maintaining its basic capabilities, and stably output high-quality game characters and ACG IP character images that meet specific task requirements such as customization.

[0041] In one embodiment of the invention, the low-rank adapter is a LoRA adapter, which consists of a pair of low-rank matrices. and The structure is as follows: where m and n represent the dimensions of the original weight matrix, and r is the rank of the low-rank matrix. The update of the original weight matrix is ​​transformed into an operation on the low-rank matrix. The rank r of the low-rank matrix satisfies: 4≤r≤64.

[0042] In this invention, parameter updates are applied only to the weight matrix of the cross-attention layer of the basic Stable Diffusion model. This reduces the number of parameters and computational cost, improves fine-tuning efficiency, and reduces the risk of overfitting.

[0043] If the weight matrix of the cross-attention layer of the basic Stable Diffusion model is The weights adjusted by the LoRA adapter can be expressed by the following formula:

[0044]

[0045] In the formula, This indicates the weights adjusted by the LoRA adapter; Typically initialized to... Equal values, and remain unchanged or dynamically adjusted during training. Its function is to control the degree of influence of the LoRA adapter on the original model. When the LoRA adapter is small, the adjustment range is small, which helps with stability in the early stages of training; as training progresses, it can be adjusted according to the actual situation. This is to balance the model's ability to learn from new data with its retention of existing knowledge.

[0046] In this invention, the basic Stable Diffusion model has been pre-trained on a large-scale dataset, learning rich general image generation knowledge, such as color distribution patterns, object shape features, and texture patterns. In scenarios involving fine-tuning with few samples, directly updating these weights could lead to the model rapidly forgetting the general knowledge learned during pre-training due to the limitations of small sample data. This could result in catastrophic forgetting, causing a significant drop in the quality of generated images, or even the loss of the original generation capability. Therefore, this invention freezes the original weights of the basic Stable Diffusion model.

[0047] The freezing operation of the original weights of the base Stable Diffusion model includes setting the weights of all linear layers of the base Stable Diffusion model to an untrainable state, and keeping only the parameters of the low-rank adapter as trainable.

[0048] This operation severs the gradient propagation path between the base model parameters and the loss function. During backpropagation, gradients do not flow through the weights of the base Stable Diffusion model, ensuring these parameters remain unchanged during fine-tuning. Only the LoRA adapter's parameters are allowed to receive gradients and be updated. In this way, the pre-training knowledge of the base Stable Diffusion model is preserved, while the LoRA adapter focuses on learning the data patterns and features specific to the small training set, enabling efficient fine-tuning of the model on specific tasks.

[0049] After completing the initialization of the LoRA adapter and freezing the weights of the underlying Stable Diffusion model, based on the few-sample training set... The LoRA adapter is trained. The training objective is to obtain a preliminary customized model by minimizing the image reconstruction loss function. The image reconstruction loss function measures the difference between the model-generated image and the real image. During training, the parameters of the LoRA adapter are updated and adjusted using the backpropagation algorithm to reduce the value of the image reconstruction loss function, making the model-generated image as close to the real image as possible.

[0050] S3: Combine the small sample training set with the auxiliary dataset into a mixed training dataset, where the auxiliary dataset is an auxiliary dataset related to the image generation task.

[0051] In scenarios involving fine-tuning with few samples, relying solely on a small training set can easily lead to insufficient feature learning by the model, resulting in overfitting or unstable generation performance. Therefore, constructing a hybrid training dataset, merging the small training set with an auxiliary dataset, can improve model performance.

[0052] The construction of the auxiliary dataset includes:

[0053] Auxiliary data is obtained from data sources related to the image generation task, including but not limited to public image libraries, historical generated image data, and pre-trained model feature data.

[0054] Public image repositories can include large public datasets such as ImageNet and COCO, as well as various specialized art and anime image repositories. These repositories contain a vast amount of image resources, covering a rich variety of visual styles, themes, and scenes. For example, when performing a task to generate images in a two-dimensional style, a large number of images with similar styles can be obtained from specialized two-dimensional image repositories as auxiliary data. By searching public image repositories, images that are similar to the small sample training set in terms of visual style, composition characteristics, etc., can be obtained, providing the model with more reference information and helping the model learn richer image features and patterns.

[0055] Historical generated image data reflects the model's output at different stages or under different conditions, containing features and style elements relevant to the current small training set. Incorporating this historical generated image data into the auxiliary dataset helps the model better maintain and strengthen specific style characteristics, improving the coherence and consistency of generated images.

[0056] Pre-trained models learn rich general image features and semantic information on large-scale data. Feature data generated by the pre-trained model during image processing, such as feature vectors from intermediate layers and attention maps, are extracted as auxiliary data. This feature data describes the content and structure of the image from a higher-level semantic perspective, helping the current fine-tuned model better understand the semantic information of the image and establish a more accurate mapping between text prompts and image generation. For example, by using a ResNet model pre-trained on large-scale natural images to extract high-level semantic features, and combining these features with a small sample training set, the model can more accurately grasp the overall structure and semantic content of the image when generating it.

[0057] The acquired auxiliary data undergoes deduplication and image quality filtering to obtain the auxiliary dataset. Since auxiliary data obtained from multiple data sources may contain duplicate images, these duplicates not only increase data storage and computation costs but may also cause the model to overlearn on certain samples during training, affecting the model's generalization ability. Therefore, this invention requires deduplication of the auxiliary data. Specifically, deduplication methods can employ image hashing algorithms, such as perceptual hashing, average hashing, and differential hashing. These algorithms characterize image features by calculating the hash value of the image and then comparing the differences between the hash values ​​of different images (usually measured using Hamming distance). When the difference is less than a certain threshold, these images are considered duplicates and are removed. Deduplication ensures that each sample in the auxiliary dataset is unique, improving data effectiveness and training efficiency.

[0058] Meanwhile, image quality directly impacts model training performance. Low-quality images may contain noise, blur, and distortion, interfering with the model's learning of effective features and even leading to incorrect patterns. Therefore, image quality filtering is necessary for auxiliary data. Image quality can be evaluated from multiple dimensions, including sharpness, integrity, and resolution.

[0059] After the above acquisition and processing steps, the resulting auxiliary dataset and the few-sample training set are combined into a mixed training dataset. By introducing the auxiliary dataset, the model can learn more diverse features and patterns during training, enhancing its generalization ability. At the same time, with the help of regularization constraints, it effectively prevents catastrophic forgetting during few-sample fine-tuning, improving the quality and stability of the images generated by the model.

[0060] S4: Load the primary custom model as the reference model and create a complete copy of the primary custom model as the current fine-tuning model.

[0061] The reference model is Its parameters are The current fine-tuning model is Its parameters are ,at this time In the initial state, the reference model and the current fine-tuning model are completely identical in structure and parameters.

[0062] In this embodiment of the invention, a reference model and a current fine-tuned model are set to achieve the optimization goal of preventing catastrophic forgetting. The reference model retains the parameters and capabilities of the initial customized model in its current state, serving as a "benchmark." During subsequent training, a catastrophic forgetting prevention regularization loss term is calculated by comparing the output features of the reference model and the current fine-tuned model on samples from the mixed training dataset. If the output features of the current fine-tuned model differ significantly from those of the reference model during the learning of new data (the mixed training dataset), it indicates that the model may have experienced catastrophic forgetting, i.e., overlearning new data and losing previously learned knowledge. By calculating and incorporating this loss into the joint loss function, the parameter updates of the current fine-tuned model can be constrained, allowing it to adapt to new data while retaining its original capabilities to the greatest extent possible, thus avoiding catastrophic forgetting.

[0063] like Figure 4 As shown, S5: For the mixed training dataset Each sample in Perform dual-path forward computation, extracting reference output features through the reference model and extracting current output features through the current fine-tuning model.

[0064] Dual-path forward computation effectively monitors whether catastrophic forgetting occurs during model training by comparing the output features of the reference model and the currently fine-tuned model. Reference model It retains the parameters and capabilities of the initial customized model at a specific stage, representing the knowledge learned by the model in the early stages of training; the current fine-tuned model Then, as training progresses, the parameters are continuously updated, and the model learns from the mixed training dataset. The new features. By comparing the output features of the two for the same input sample, if the difference is too large, it indicates that the current fine-tuned model may have lost its original knowledge when learning new data. At this time, it is necessary to constrain the model training by using the regularization loss term to prevent catastrophic forgetting, so as to ensure that the model retains its original capabilities to the greatest extent while adapting to new data.

[0065] The reference output features and the current output features are both latent space feature vectors of the image, which are extracted through the decoder output layers of the reference model and the current fine-tuned model.

[0066] Specifically, in the forward computation of the reference model, firstly, the text prompts are... Input reference model Encoder via reference model The text is encoded, converting the natural language text prompts into feature vector representations that the model can process. These feature vectors then enter a cross-attention layer, interacting with other relevant features to establish associations between text and image features. Finally, the text passes through a decoder. The encoded feature vector is decoded into an image latent space feature vector to obtain the reference output feature. ,Right now .

[0067] In the fine-tuning of the model's forward computation, text prompts will be used. Input fine-tuning model After passing through the same encoder as the reference model Cross-attention layer and decoder Extract the current output features ,Right now .

[0068] Because the parameters of the current fine-tuning model are constantly updated during training, it can respond to the same text prompts... The processing results may differ from the reference model, and this difference reflects how the model changes during the learning process of the mixed training dataset.

[0069] Furthermore, it should be noted that in the specific embodiments of this invention, to ensure the accuracy and stability of the calculation, the input data also needs to be standardized, such as segmenting and vectorizing the text prompts, and normalizing the image data. Simultaneously, the latent space feature vectors output by the model also require appropriate post-processing, such as adjusting the dimensions and normalizing, to facilitate feature comparison and loss calculation. In the Stable Diffusion model, the latent space feature vectors are low-dimensional representations containing key information such as shape, color, and texture required to generate the image, and can be transformed into an actual image through image generation operations.

[0070] like Figure 5 As shown, S6: Based on reference output features With current output features Calculate the loss term for preventing catastrophic forgetting based on the distribution differences between them. This quantifies the degree of deviation between the current fine-tuned model and existing knowledge when learning new data, thereby constraining the model training process and ensuring that the model retains the learned capabilities to the greatest extent while adapting to new data.

[0071] Among them, reference output features With current output features The distributional differences between them can be measured by the KL divergence, which is calculated using the following formula:

[0072]

[0073] In the formula, K represents the dimension of the latent space feature vector of the image, which determines the number and complexity of features. In the Stable Diffusion model, this dimension is determined by the model architecture and design and is set to a fixed value. For example, K is set to 339, which represents the number of features of the image after being encoded in the latent space.

[0074] and They are respectively and The Each element represents a specific value of the latent space feature vector of the image in each dimension. These values ​​reflect the information representation of the image in different feature dimensions.

[0075] For mixed training datasets Each sample One can be calculated The entire mixed training dataset The loss term is a regularization term to prevent catastrophic forgetting. It is expressed as the average of all samples, and its calculation formula is as follows:

[0076]

[0077] In the formula, For mixed training datasets By averaging the sample size, we can obtain a comprehensive loss value that reflects the model's knowledge retention across the entire dataset.

[0078] like Figure 6 As shown, S7: Based on a mixed training dataset The current fine-tuned model calculates the image reconstruction loss function. Specifically, the image reconstruction loss function calculation includes:

[0079] Mixed training datasets Text prompts in Input to the current fine-tuning model The model converts text into semantic feature vectors through an encoder, establishes a correlation between text and image features through a cross-attention layer, and finally generates the corresponding image through a decoder. Image generation based on model With mixed training datasets Real images The difference is used to calculate the image reconstruction loss function. .

[0080] Among them, the image reconstruction loss function The following conditions must be met:

[0081] When comparing the generated image with the real image in pixel space, the mean squared error (MSE) loss function for the RGB channels is used. This function quantifies the differences between the generated image and the real image in terms of intuitive visual aspects such as color and shape by calculating the sum of squared errors for each pixel across the three RGB channels and averaging the results. The formula is:

[0082]

[0083] In the formula, and These represent the width and height of the image, respectively. Indicates the RGB channel; and These represent the generated image and the real image at the 1st and 2nd respectively. aisle, The MSE loss function assigns the same weight to all pixels, prompting the model to generate images that are as close as possible to the real image during training. This is suitable for tasks that require detailed reproduction.

[0084] When comparing the generated image with the real image in the latent space, the L1 norm loss function for latent variable features is used. In the Stable Diffusion model, the image is encoded into a low-dimensional latent space feature vector before generation. This vector contains high-level information such as semantics and structure. The L1 norm loss function measures the difference between the generated image and the real image at the semantic and structural levels by averaging the sum of the absolute values ​​of the differences between corresponding elements of the latent space feature vectors. Its formula is:

[0085]

[0086] In the formula, This represents the dimension of the latent space eigenvectors. and They are respectively and The Each element. Compared to the MSE loss function, the L1 norm loss function is more robust to outliers, pays more attention to the overall structure and semantic consistency of the image, and is suitable for tasks with high requirements for image style and semantic expression.

[0087] Combined with the calculated image reconstruction loss function and prevent catastrophic forgetting regularization loss terms According to the preset weighting coefficients for the loss term to prevent catastrophic forgetting, Constructing a joint loss function by addition ,in, .

[0088] The weight coefficient of the regularization loss term to prevent catastrophic forgetting is set to Q1 in the early stage of training and Q2 in the later stage of training, where Q1>Q2.

[0089] For example, the weighting coefficients of the loss term to prevent catastrophic forgetting regularization. The value can be set to 0.8 in the first 30% of training rounds and to 0.3 in the latter 70% of training rounds.

[0090] In the early stages of training, a larger λ value can strengthen the suppression of catastrophic forgetting and ensure model stability. As training progresses, reducing the λ value allows the model to learn new data features more fully, resulting in performance improvements. The joint loss function balances these two objectives, guiding the model to achieve optimal training results during fine-tuning with small samples.

[0091] In this embodiment of the invention, the current fine-tuning model is updated through backpropagation. The updated low-rank adapter parameters include the weight parameters of the low-rank matrix pairs in the cross-attention layer. Furthermore, the updated low-rank adapter parameters also include bias term parameters associated with the low-rank matrix pairs. During the update process, the original weight gradients of the base Stable Diffusion model are masked; that is, only the parameters of the low-rank adapter are updated, avoiding overfitting and catastrophic forgetting problems caused by directly updating the parameters of the base Stable Diffusion model.

[0092] Among them, the backpropagation algorithm calculates... For low-rank adapter parameters gradient Determine the direction of parameter updates and combine it with the learning rate. Update the parameters using the following formula:

[0093]

[0094] In the formula, This indicates the number of training iterations.

[0095] like Figure 6 As shown, S8: Repeat S5 to S7 until the set termination condition is met, and then stop, and output the final updated current fine-tuned model as the final optimized model.

[0096] In one embodiment of the present invention, the termination condition is set as follows: repeating S5 to S7 until a preset maximum number of training rounds is reached. Or the decrease in the joint loss function value for K consecutive rounds is less than a preset threshold. The process will end at that time.

[0097] Specifically, during training, after each round of calculations and parameter updates (S5-S7), the joint loss function value for the current round is calculated. and the joint loss function of the previous round Compare and calculate the magnitude of the decline. ,in, The expression is:

[0098]

[0099] like If the condition is met consecutively, the counter `count` will be incremented by 1; if... If this happens, the count will be reset to 0. For example, Set to 100, K to 5. The value is set to 0.002, meaning that training stops when the joint loss function value decreases by less than 0.002 for five consecutive rounds. Simultaneously, the number of training rounds is continuously monitored during training; once 100 rounds are reached, training will immediately terminate regardless of the count value.

[0100] By setting the aforementioned dual termination conditions, the model is ensured to have sufficient training epochs to learn data features, while training is stopped promptly when the model's performance improvement plateaus. The final optimized model avoids catastrophic forgetting and effectively utilizes small sample data and auxiliary datasets for performance optimization. In practical application scenarios (such as game characters, customized anime-style IPs, etc.), it can stably generate high-quality images that meet customization requirements, demonstrating the algorithm's efficiency and practicality in small sample fine-tuning tasks.

[0101] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. "A plurality of" means two or more, unless otherwise explicitly specified.

[0102] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0103] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0104] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0105] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0106] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0107] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0108] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0109] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0110] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A small-sample fine-tuning method based on regularization constraints to prevent catastrophic forgetting, characterized in that, Includes the following steps: S1: Obtain the basic Stable Diffusion model and the few-shot training set; wherein, the few-shot training set contains N sets of text-image pairs; S2: Initialize the low-rank adapter and insert it into the cross-attention layer of the base Stable Diffusion model; freeze the original weights of the base Stable Diffusion model; and train the low-rank adapter based on the small sample training set to obtain a primary customized model by minimizing the image reconstruction loss function. S3: Merge the small sample training set and the auxiliary dataset into a mixed training dataset, wherein the auxiliary dataset is an auxiliary dataset related to the image generation task; S4: Load the primary customized model as a reference model, and create a complete copy of the primary customized model as the current fine-tuning model; S5: Perform dual-path forward computation on each sample in the mixed training dataset, extract reference output features through the reference model, and extract current output features through the current fine-tuning model; S6: Calculate the catastrophic forgetting regularization loss term based on the distribution difference between the reference output features and the current output features; S7: Calculate the image reconstruction loss function based on the hybrid training dataset and the current fine-tuning model; combine the calculated image reconstruction loss function and the catastrophic forgetting regularization loss term, and construct a joint loss function by adding them according to the preset weight coefficients of the catastrophic forgetting regularization loss term, wherein the weight coefficients of the catastrophic forgetting regularization loss term are set to Q1 in the early stage of training and Q2 in the later stage of training, wherein Q1>Q2; update the low-rank adapter parameters in the current fine-tuning model through backpropagation; S8: Repeat S5 to S7 until the set termination condition is met, and then stop and output the current fine-tuned model as the final optimized model.

2. The small-sample fine-tuning method for preventing catastrophic forgetting based on regularization constraints according to claim 1, characterized in that, In S2, the low-rank adapter is a LoRA adapter, which consists of a pair of low-rank matrices, and the rank r of the low-rank matrix satisfies: 4≤r≤64; its parameter update only applies to the weight matrix of the cross-attention layer of the basic Stable Diffusion model.

3. The small-sample fine-tuning method for preventing catastrophic forgetting based on regularization constraints according to claim 2, characterized in that, In step S2, the freezing operation of the original weights of the basic Stable Diffusion model includes: Set the weights of all linear layers in the base Stable Diffusion model to an untrainable state, and retain only the parameters of the low-rank adapter as trainable.

4. The small-sample fine-tuning method for preventing catastrophic forgetting based on regularization constraints according to claim 3, characterized in that, In step S3, the construction of the auxiliary dataset includes: Auxiliary data is obtained from data sources related to the image generation task, including but not limited to public image libraries, historical generated image data, and pre-trained model feature data; The auxiliary dataset is obtained by deduplicating the acquired auxiliary data rows and filtering the image quality.

5. The small-sample fine-tuning method for preventing catastrophic forgetting based on regularization constraints according to claim 4, characterized in that, In step S5, both the reference output feature and the current output feature are image latent space feature vectors, which are extracted through the decoder output layer of the reference model and the current fine-tuning model.

6. The small-sample fine-tuning method for preventing catastrophic forgetting based on regularization constraints according to claim 5, characterized in that, In step S6, the distribution difference between the reference output feature and the current output feature is measured by KL divergence.

7. The small-sample fine-tuning method for preventing catastrophic forgetting based on regularization constraints according to claim 6, characterized in that, In step S7, the calculation of the image reconstruction loss function includes: inputting the text prompts in the mixed training dataset into the current fine-tuning model to obtain the corresponding model-generated image; and calculating the image reconstruction loss function based on the difference between the model-generated image and the real image in the mixed training dataset. The image reconstruction loss function satisfies the following condition: When comparing the model-generated image with the real image in pixel space, the mean squared error loss function of the RGB channels is used; When comparing the model-generated image with the real image in the latent space, the L1 norm loss function of the latent variable features is used.

8. The small-sample fine-tuning method for preventing catastrophic forgetting based on regularization constraints according to claim 7, characterized in that, In step S7, the updated low-rank adapter parameters include: Weight parameters of low-rank matrix pairs in the cross-attention layer; The bias term parameters associated with the low-rank matrix pair; and the original weight gradients of the underlying Stable Diffusion model are masked during the update process.

9. The small-sample fine-tuning method for preventing catastrophic forgetting based on regularization constraints according to claim 8, characterized in that, In S8, the termination condition is set as follows: S5 to S7 are repeated until the preset maximum number of training rounds is reached or the value of the joint loss function decreases less than a preset threshold for K consecutive rounds.

Images