Small-sample progressive fine-grained multi-label model training method, electronic equipment and storage medium

Abstract

The invention discloses a small-sample progressive fine-grained multi-label model training method. The method comprises the following steps: firstly, systematically solving the problems of scarcity of vertical data and complicated concepts by constructing a hierarchical label system and a differentiated data strategy; furthermore, a progressive three-stage training method is adopted, and domain features are quickly learned on a small number of samples. And finally, outputting an available model with professional generation capability and good generalization.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image generation, in particular to a small sample progressive fine-grained multi-label model training method, an electronic device and a storage medium. BACKGROUND

[0002] In recent years, text-to-image large models based on diffusion models (such as Stable Diffusion, Stable Diffusion XL, Flux, etc.) have made significant progress and can generate high-quality and diversified images according to natural language descriptions. These models are mainly trained on general large-scale datasets (such as LAION), and their generated styles tend to be realistic or general digital art styles. However, when these general models are applied to specific vertical fields (such as art illustration, medical imaging, industrial design, specific animation styles, etc.), the generation effect will deviate significantly. Taking the art illustration field as an example, the existing models generally have the following problems: the color saturation of the generated images is high, the contrast is strong, and there are obvious "AI generation traces" (such as texture abnormalities and unreasonable details), which are far from the expected hand-drawn quality, soft color tone, and artistic style, and cannot meet the needs of professional commercial applications.

[0003] To adapt to vertical fields, the industry usually adopts the method of retraining (fine-tuning) the pre-trained large model. However, this process faces three major challenges: (1) Data scarcity: High-quality, large-scale vertical field data is extremely costly to collect and label, and often involves copyright issues, resulting in extremely scarce training samples.

[0004] (2) Concept complexity: There are numerous professional concepts in vertical fields, and the label system is complex and fine-grained (for example, "watercolor blending", "pencil sketch line direction", "cellophane style hatching", etc.), making it difficult to accurately describe with general labels.

[0005] (3) Training paradox: A long training period can cause the model to overfit (Overfitting), thereby forgetting the strong generalization ability of the pre-trained model; while a short training period cannot allow the model to fully learn the subtle features of the vertical field, resulting in insufficient learning.

[0006] Existing technologies (such as LoRA, DreamBooth, etc.) can achieve small sample learning to some extent, but have not systematically solved the balance between fine-grained multi-concept learning and generalization preservation. Therefore, there is an urgent need for an innovative method that can quickly and accurately learn multiple fine-grained concepts in vertical fields under the condition of a small number of samples, while maximizing the preservation of the original capabilities of the base model. SUMMARY

[0007] The main purpose of the present application is to provide a small sample progressive fine-grained multi-label model training method, electronic equipment and storage medium, solve the problem of data shortage, complex concept and training paradox in the process of vertical field text-to-image model training; this method can realize fast convergence and accurate learning of fine-grained visual concept and style in the field under the condition of very small amount of training samples, while significantly maintaining the original generalization ability of the model.

[0008] In order to achieve the above purpose, one of the solutions of the present application is: A small sample progressive fine-grained multi-label model training method, comprising: Step 1: For the target vertical field, a structured label system is constructed, which not only contains the description of regular objects and actions, but also contains style and attribute labels; this label system is divided into three levels: (1) Macro style layer, used to define the overall style of the target vertical field; (2) Mid-level technique layer, used to describe the specific techniques for realizing the style; (3) Micro parameter layer, used to describe the quantifiable visual parameters; Step 2: Using the label system in step 1, pre-process the training images, label multiple labels from different levels of the label system for each training image, and construct an image dataset; Step 3: Adopting a coarse-to-fine progressive training strategy, fine-tune the pre-trained text-to-image model in three stages: Step 3.1: Use all training images, and use only the macro style layer label for text input to train the model in the first stage; in terms of training parameters, the first stage uses a learning rate of 1e-5 to 1e-4, freezes 80% to 90% of the network parameters of the model, and only trains the text encoding layer and cross attention layer; Step 3.2: Use all training images, and use all levels of labels in the label system for text input to train the model in the second stage; in terms of training parameters, the second stage increases the learning rate to 5e-4, unfreezes 95% of the network layers or all network layers for training, and continues training based on the first stage; Step 3.3: Mix all vertical field and 5% general field images and labeled texts, and use all levels of labels for text input to train the model in the third stage; in terms of training parameters, the third stage uses a learning rate of 1e-5 to 1e-4 to train all network parameters; Step 4: Model output and deployment After training is completed, a text-to-image model adapted to the target vertical field is obtained; the user can generate target images by inputting text prompt words.

[0009] Preferably, the step 2 comprises: Step 2.1: Collect a relatively large batch of images to construct a macro dataset; the core requirement of the batch of images is that the overall style is clear and consistent; label the macro style layer for the batch of images; Step 2.2: Select a subset from the macro dataset of step 2.1, or additionally collect a relatively moderate batch of images to construct a meso dataset; the core requirement of the batch of images is to show the techniques defined in step 1, and one image contains at least one technique; label the meso technique layer for the batch of images; Step 2.3: Select a subset from the macro dataset of step 2.1 and / or the meso dataset of step 2.2, or additionally collect a relatively small batch of images to construct a micro dataset; the core requirement of the batch of images is that the parameters meet the professional requirements of the vertical field; accurately label the micro parameter layer for the batch of images.

[0010] The second solution of the present application is: An electronic device comprising a processor, a memory and an application program; the application program is stored in the memory and is configured to execute the small sample progressive fine-grained multi-label model training method by the processor.

[0011] The third solution of the present application is: A computer readable storage medium, the storage medium stores a computer program; the computer program executes in the computer, makes the computer execute the small sample progressive fine-grained multi-label model training method.

[0012] After adopting the above technical solution, the present application has the following technical effects: Firstly, by constructing a fine-grained hierarchical label system, the present application can accurately describe complex and fine professional concepts in the vertical field, effectively solving the problem of inaccurate general label description. This system not only covers the basic concepts in the field, but also further refines the granularity of the concepts, so that the model can more accurately capture and learn the visual features specific to the field.

[0013] Secondly, the introduction of differentiated small sample training set construction and preprocessing technology greatly alleviates the problem of data scarcity. The present application carefully selects and preprocesses a small amount of high-quality samples to ensure the effectiveness and representativeness of the training data, thereby achieving efficient model training under limited samples.

[0014] Furthermore, the implementation of the progressive three-stage training strategy successfully balanced the relationship between fine-grained multi-concept learning and generalization preservation. In the macro-style alignment stage, the model initially adapts to the overall style of the vertical domain; in the fine-grained concept injection stage, it delves into learning specific concepts and details within the domain; finally, in the model generalization and vertical concept reconciliation stage, the model maintains its learning effect on the vertical domain while also preserving the original generalization ability of the base model to the greatest extent. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the steps of a specific embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the labeling system in step 1 of a specific embodiment of the present invention.

[0017] Figure 3 This is a flowchart of step 2 in a specific embodiment of the present invention.

[0018] Figure 4 This is a flowchart of step 3 in a specific embodiment of the present invention. Detailed Implementation

[0019] To further explain the technical solution of the present invention, the present invention will be described in detail below through specific embodiments.

[0020] Reference Figures 1 to 4 As shown, this invention discloses a small-sample, progressive, fine-grained, multi-label model training method. First, by constructing a hierarchical labeling system and a differentiated data strategy, it systematically overcomes the challenges of scarce vertical category data and complex concepts. Then, a progressive three-stage training method is employed to rapidly learn domain features on a small number of samples. Finally, a usable model with both professional generation capabilities and good generalization ability is output. See also... Figure 1 Specifically, the training method includes: Step 1: Construct a fine-grained hierarchical tagging system See Figure 2 For the target vertical domain, a structured tagging system is constructed, which not only includes conventional object and action descriptions (such as "a samurai" or "holding a weapon"), but more importantly, also includes multi-level, fine-grained style and attribute tags; this tagging system is divided into three levels: (1) Macro style layer, used to define the overall style of the target vertical field, such as Japanese cel animation, European and American cartoons, Chinese ink painting style, etc.; (2) The mid-level technique layer describes the specific techniques used to achieve the style, such as flat painting, thick painting, outlining, smudging, noise reduction, etc. (3) Micro-parameter layer, used to describe quantifiable visual parameters, such as low saturation, high brightness, low contrast, soft shadows, soft colors, etc.

[0021] Therefore, by subsequently labeling each training image with multiple labels from different levels of the labeling system, rich text descriptions can be constructed.

[0022] Step 2: Construction and preprocessing of differentiated small sample training sets The purpose of step 2 is to collect a small-scale but internally structured image dataset from a vertical domain. Its key innovation lies in employing differentiated data collection and annotation strategies based on the labeling system established in step 1 to maximize the efficiency of utilizing limited data. Specifically, see [link to relevant documentation]. Figure 3 Using the labeling system from step 1, the training images are preprocessed to label each training image with multiple labels from different levels of the labeling system, thus constructing an image dataset.

[0023] Furthermore, the detailed steps of step 2 above include: Step 2.1: Macro-style layer data collection (basic dataset, largest quantity) A relatively large batch of images (e.g., approximately 1000 images per macro style) is collected to construct a macro dataset. The core requirement for this batch of images is that the overall style is clear and consistent, while the requirements for the diversity of image details and specific techniques can be appropriately relaxed. These images are primarily labeled with macro style layer tags, such as Japanese cel animation, Western cartoons, and Chinese ink painting style. This allows for the construction of a solid style foundation, aiming to broadly cover the overall style of the target vertical domain, ensuring that the model can learn the overall distribution and commonalities of styles.

[0024] Step 2.2: Data collection at the meso-level techniques (core dataset, moderate quantity) From the macro dataset in step 2.1, select a subset, or collect an additional batch of images of a suitable size (e.g., about 200 images for each meso technique) to construct a meso dataset. The core requirement for this batch of images is that they clearly demonstrate the techniques defined in step 1, and each image contains at least one technique. Provide detailed annotations for this batch of images, adding labels for the meso technique layer, such as flat painting, thick painting, outlining, shading, noise reduction points, etc. This is the core of training, ensuring that the model can accurately cover the key techniques and concepts that need to be learned under different macro styles.

[0025] Step 2.3: Collection of microscopic parameter layer data (calibration dataset, small in quantity) From the macro dataset in step 2.1 and / or the meso dataset in step 2.2, select a subset, or additionally collect a relatively small batch of images (e.g., about 100 images) to construct a micro dataset. The core requirement for this batch of images is that they perfectly meet the professional requirements of the vertical field in terms of parameters such as color, tone, and texture. Accurately label this batch of images with micro parameter layers, such as low saturation, high brightness, low contrast, soft shadows, and soft colors. Although this part of the data is small, it acts as a "calibration sample" and is crucial for correcting the underlying output of the model, ensuring that the model can accurately control the underlying visual attributes of the generated images and solve deviations in color, contrast, etc.

[0026] Step 3: Progressive Three-Stage Training Strategy See Figure 4 A progressive training strategy, from coarse to fine, is adopted to fine-tune the pre-trained text-to-image model in three stages: Step 3.1: Macro Style Alignment Using all training images, and text input (prompt words) labeled only with macro-style layers (e.g., "a picture with a Japanese cel-shaded style"), the model is trained in the first stage. The training goal is to enable the model to initially learn the overall style distribution of the vertical domain. In terms of training parameters, a low learning rate (e.g., 1e-5 to 1e-4) is used in the first stage to freeze most (e.g., 80% to 90%) of the network parameters of the model, and only the text encoding layer and cross-attention layer are trained to prevent catastrophic forgetting.

[0027] Step 3.2: Fine-grained concept injection Using all training images and text input (prompt words) with labels from all levels of the labeling system, the model is trained in the second stage. The training goal is to enable the model to learn more refined techniques and attribute concepts based on style alignment. In terms of training parameters, the learning rate is appropriately increased in the second stage (e.g., increased to 5e-4), and more (e.g., 95%) of the network layers or all network layers are unfrozen and trained together for deeper fine-tuning.

[0028] Step 3.3: Model Generalization and Reconciliation of Vertical Concepts By using a mix of vertical domain data and a small amount (e.g., 5%) of general data (sampled from the original pre-training data), and with text input (prompt words) containing labels from all levels, the model undergoes a third stage of training. The training objective is to prevent the model from overfitting to the small training set and to restore and strengthen its generalization ability. In terms of training parameters, the third stage uses a low learning rate (e.g., 1e-5 to 1e-4) to train all parameters of the network, prompting the model to find the optimal balance between vertical style and general knowledge.

[0029] Step 4: Model Output and Deployment After training, a text-based image model adapted to the target vertical domain is obtained; users can generate high-quality images that meet the domain's professional requirements by inputting text prompts containing fine-grained descriptions.

[0030] Through the above solution, the present invention can achieve the following technical effects: (1) Efficient data utilization and knowledge mining: By leveraging the "fine-grained hierarchical labeling system", the potential value of a small amount of data is fully explored, which solves the problems of scarce vertical data and complicated concept labeling.

[0031] (2) Ingenious training process design: Fine-grained labels are used to guide the model to learn professional concepts. Combined with a progressive learning strategy of style alignment, concept injection and generalization harmonization, the model can master complex styles and techniques from shallow to deep. This "progressive three-stage training strategy" ensures that the learning process is gradual and steady, while achieving an excellent balance in terms of efficiency (time, computing power) and effectiveness (quality, generalization).

[0032] (3) Excellent vertical domain adaptability: The generated images are more professional, accurate and artistic, and can effectively eliminate the "incompatibility" of general models (such as color oversaturation, unnatural texture and other problems).

[0033] (4) Strong practicality and commercial value: The solution fully considers the cost, efficiency and model flexibility in the actual deployment process, and is suitable for quickly customizing high-quality text graph models for different vertical fields, thereby reducing the threshold for commercialization.

[0034] Furthermore, the present invention also discloses an electronic device including a processor, a memory, and an application program; the application program is stored in the memory and configured to be executed by the processor using the methods described above.

[0035] The present invention also discloses a computer-readable storage medium storing a computer program; when the computer program is executed in a computer, it causes the computer to perform the above-described method. The storage medium may include various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0036] The above embodiments and figures are not intended to limit the product form and style of the present invention. Any appropriate changes or modifications made by those skilled in the art should be considered as not departing from the patent scope of the present invention.

Claims

1. A method for small-sample progressive fine-grained multi-label model training, characterized in that, include: Step 1: For the target vertical domain, construct a structured tagging system that includes not only regular object and action descriptions but also style and attribute tags; this tagging system is divided into three levels: (1) Macro style layer, used to define the overall style of the target vertical field; (2) The mid-level technique layer, used to describe the specific techniques used to achieve the style; (3) Micro-parameter layer, used to describe quantifiable visual parameters; Step 2: Using the labeling system from Step 1, preprocess the training images, label each training image with multiple labels from different levels of the labeling system, and construct an image dataset; Step 3: Employ a coarse-to-fine progressive training strategy, fine-tuning the pre-trained text-to-image model in three stages: Step 3.1: Use all training images and only use the labels of the macro style layer for text input to train the model in the first stage; Regarding training parameters, the first stage uses a learning rate of 1e-5 to 1e-4, freezes 80% to 90% of the network parameters of the model, and trains only the text encoding layer and the cross attention layer. Step 3.2: Use all training images and text inputs with labels from all levels of the labeling system to train the model in the second stage; Regarding training parameters, in the second stage, the learning rate is increased to 5e-4, 95% of the network layers are unfrozen or all network layers are trained together, and training continues based on the first stage. Step 3.3: Use a mix of images and labeled text from all vertical domains and 5% of the general domain. The text input includes labels from all levels. Train the model in the third stage. In terms of training parameters, the third stage uses a learning rate of 1e-5 to 1e-4 to train all parameters of the network. Step 4: Model Output and Deployment After training, a text-based image model adapted to the target vertical domain is obtained; users can generate target images by inputting text prompts.

2. The small-sample progressive fine-grained multi-label model training method as described in claim 1, characterized in that, Step 2 includes: Step 2.1: Collect a relatively large batch of images to construct a macro dataset; the core requirement for this batch of images is that the overall style is clear and consistent; label this batch of images with macro style layer labels; Step 2.2: Select a subset from the macro dataset in Step 2.1, or collect an additional batch of images of a relatively moderate size to construct a meso dataset; the core requirement for this batch of images is that they demonstrate the techniques defined in Step 1, and each image contains at least one technique; label this batch of images with meso technique layer labels; Step 2.3: Select a subset from the macro dataset in Step 2.1 and / or the meso dataset in Step 2.2, or collect a relatively small number of images to construct a micro dataset; the core requirement for this batch of images is that the parameters meet the professional requirements of the vertical field; accurately label the micro parameter layers of this batch of images.

3. An electronic device, characterized in that... It includes a processor, memory, and an application program; the application program is stored in the memory and configured to be executed by the processor as described in claim 1 or 2, using the few-shot progressive fine-grained multi-label model training method.

4. A computer-readable storage medium, characterized in that... The storage medium stores a computer program; when the computer program is executed in the computer, it causes the computer to perform the small-sample progressive fine-grained multi-label model training method as described in claim 1 or 2.

Images