Method and system for generating similar images by using LoRA model

Abstract

The invention discloses a method and system for generating similar images by using a LoRa model, and is applied to the technical field of artificial intelligence, and the method comprises the steps: 1, loading a pre-training model; step 2, configuring a LoRa matrix; step 3, applying a LoRa matrix to a pre-training model; 4, preparing a training data set; step 5, training the LoRa model; and step 6, generating a similar image. According to the method, the LoRa model is utilized to generate the similar image, the method has the characteristics of high efficiency and flexibility, and only a small number of parameters in the pre-training model need to be adjusted, so that the demand of computing resources and the training time are greatly reduced; a user can adjust parameters and input parameters of the LoRa matrix according to needs so as to generate similar images with different styles and characteristics; meanwhile, due to the introduction of the LoRa matrix, the model can generate high-quality similar images while keeping the performance of the original pre-training model; and the method can be conveniently integrated into various image generation systems.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for generating similar images using a LoRa model. Background Art

[0002] Similar images refer to images that have visually similar features or content. These images may be similar in color, texture, shape, layout, or overall theme. Similar images may come from different shooting angles, lighting conditions, resolutions, or processing techniques, but the information they convey or the themes they present are often similar or identical. In the field of computer vision, for pattern recognition and machine learning, if there are a large number of similar images as samples, it can improve the generalization ability and robustness of the model, increase the diversity of the model's training samples, and improve the model performance.

[0003] With the development of deep learning technology, image generation has become an important research direction in the field of computer vision. However, fully fine-tuning a pre-trained model to generate similar images of a specific style often requires a lot of computing resources and time. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art, solve the problems of large resources required for calculating and generating similar images, low efficiency, poor flexibility and difficulty in implementation, and provide a method and system for generating similar images using a LoRa model.

[0005] The objective of the present invention is achieved through the following technical solutions: A method for generating a similar image using a LoRa model comprises the following steps: Step 1: Load the pre-trained model: Load the required pre-trained model based on the image generation task; Step 2: Configure the LoRa matrix: According to the importance of each layer in the pre-trained model and the requirements of the image generation task, clearly define the target layer where the LoRa matrix needs to be applied, and define the LoRa matrix parameters; Step 3: Apply the LoRa matrix to the pre-trained model: Instantiate the LoRa matrix configuration object, pass the parameters of the LoRa matrix to the target layer of the pre-trained model through the PEFT technology, and add the LoRa matrix to the target layer; and fix the rest of the pre-trained model, allowing only the weights of the LoRa matrix to be updated during the training process; Step 4: Prepare training data set: select high-quality images similar to reference images as training set and preprocess the training set; Step 5: Train the LoRa model: Use the preprocessed training data set to train the pre-trained model containing the LoRa matrix, and adjust the weight of the LoRa matrix through the optimization algorithm to minimize the difference between the generated image and the reference image to obtain the LoRa model; Step 6. Generate similar images: Generate images similar to the reference image through the LoRa model.

[0006] Furthermore, the pre-trained model is obtained after training with a large amount of data.

[0007] Furthermore, the LoRa matrix parameters include rank, scaling factor and dropout rate.

[0008] Furthermore, the preprocessing process in step 4 includes adjusting image size, cropping and normalization.

[0009] Further, the specific process of adjusting the weight of the LoRa matrix by the optimization algorithm in the step 5 is: Step 501, define a loss function: used to evaluate the difference between the generated image and the reference image; Step 502, gradient descent optimization: Calculate the gradients of two low-rank matrices A and B in the LoRa matrix based on the loss function, and use the gradient descent method or Adam optimizer to update the weight parameters of the low-rank matrices A and B; Step 503: Repeat steps 501 and 502 until the value of the loss function reaches a preset threshold or the number of iterations reaches a preset maximum value.

[0010] Furthermore, the loss function adopts a mean square error function.

[0011] Furthermore, the training process of step 5 also includes monitoring the loss value and the quality of the generated images during the training process to ensure that the model is trained as expected.

[0012] Furthermore, the instantiated LoRa matrix configuration object is specifically: setting the LoRa class in the PEFT library to specify the task type, the target layer to which the LoRa matrix needs to be applied, defining the LoRa matrix parameters, and passing in all configuration parameters.

[0013] A system for generating similar images using a LoRa model, comprising: Pre-trained model loading module: used to load existing or trained pre-trained models; Dataset preparation module: responsible for selecting, preprocessing and annotating training images; LoRa matrix configuration module: used to define LoRa matrix parameters, instantiate LoRa matrix configuration objects, and complete the transfer of LoRa matrix parameters to the target layer of the pre-trained model; LoRa model training module: used to train the pre-trained model containing the LoRa matrix to obtain the target LoRa model; Image generation module: Use the trained LoRa model to generate an image similar to the target image based on the input conditions.

[0014] Beneficial effects of the present invention: The present invention uses the LoRa model to generate similar images with high efficiency and flexibility. Only a small number of parameters in the pre-trained model (i.e., the LoRa matrix) need to be adjusted, which greatly reduces the demand for computing resources and training time. Users can adjust the parameters and input parameters of the LoRa matrix as needed to generate similar images with different styles and characteristics. At the same time, due to the introduction of the LoRa matrix, the model can generate high-quality similar images while maintaining the performance of the original pre-trained model; and can be easily integrated into various image generation systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0016] Figure 1 It is a schematic diagram of the flow of the method for generating similar images of the present invention.

[0017] Figure 2 This is a schematic diagram of the structure of the system for generating similar images of the present invention. DETAILED DESCRIPTION

[0018] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] In this embodiment, if Figure 1 As shown, a method for generating a similar image using a LoRa model includes the following steps: Step 1: Load the pre-trained model: Load the required pre-trained model based on the image generation task, where the pre-trained model is obtained by training with a large amount of data; the model has been trained on a large amount of data and has the ability to generate high-quality images; Step 2. Configure the LoRa matrix: According to the importance of each layer in the pre-trained model and the requirements of the image generation task, clearly define the target layer where the LoRa matrix needs to be applied, and define the LoRa matrix parameters, including rank (r), scaling factor (lora_alpha), and dropout rate (lora_dropout); Step 3: Apply the LoRa matrix to the pre-trained model: Instantiate the LoRa matrix configuration object, pass the parameters of the LoRa matrix to the target layer of the pre-trained model through the PEFT technology, and add the LoRa matrix to the target layer; and fix the rest of the pre-trained model, allowing only the weights of the LoRa matrix to be updated during the training process; Step 4: Prepare the training data set: Select high-quality images similar to the reference images as the training set, and preprocess the training set. The preprocessing includes image resizing, cropping, normalization, etc. Step 5: Train the LoRa model: Use the preprocessed training data set to train the pre-trained model containing the LoRa matrix, and adjust the weight of the LoRa matrix through the optimization algorithm to minimize the difference between the generated image and the reference image to obtain the LoRa model; At the same time, monitor the loss value and the quality of generated images during training to ensure that the model is trained as expected; Step 6. Generate similar images: Generate images similar to the reference image through the LoRa model.

[0021] In this embodiment, the LoRa class is set in the PEFT library to specify the task type, the target layer to which the LoRa matrix needs to be applied, and the LoRa matrix parameters are defined; the LoRa configuration class is instantiated and all configuration parameters are passed in; the LoRa configuration is applied to the pre-trained model through functions in the PEFT library (such as get_peft_model), which will modify some layers of the model and add the LoRa matrix to them.

[0022] Among them, the parameters that need to be configured in the LoRa class are as follows: task_type: specifies the task type, such as autoregressive language modeling (CAUSAL_LM) or sequence-to-sequence task (SEQ2SEQ_LM); target_modules: specifies which layers need the LoRa matrix applied; these are usually the parts of the model responsible for key calculations, such as the query (q_proj), key (k_proj), value (v_proj), and output (o_proj) projections of the self-attention mechanism; r: The rank of the LoRa matrix, which is used to control the size of the low-rank matrix. A larger rank value will increase the expressiveness of the model, but will also increase the computational complexity; lora_alpha: scaling factor used to adjust the impact of the low-rank matrix on the model output; lora_dropout: The dropout rate applied in the LoRa layer to prevent overfitting.

[0023] In this embodiment, the target layer to which the LoRa matrix needs to be applied is clearly defined according to the importance of each layer in the pre-trained model and the image generation task requirements: (1) Judging by the number of parameters and the function of the layer, layers with more parameters contain more information, and fine-tuning these layers may have a more significant impact on model performance; different layers have different functions in the model (for example, in GAN (generative adversarial network) , The last few layers of the generator are usually responsible for generating image details), so fine-tuning these layers may be more helpful in improving the quality of generated images; (2) Image generation task requirements: Select appropriate layers for fine-tuning based on the requirements of the specific task. For example, if the task requires improving the diversity of generated images, then it is necessary to fine-tune the layers in the generator that are related to image diversity. Before deciding which layer to fine-tune, you can conduct some preliminary experiments. By fine-tuning different layers and observing the changes in model performance, you can preliminarily determine which layers have a greater impact on model performance; and use statistical methods such as cross-validation to evaluate the effects of different fine-tuning strategies; This helps to more accurately determine which layer is most suitable for LoRA fine-tuning.

[0024] In this embodiment, the weights in the LoRa (Low-Rank Adaptation) matrix are adjusted with the goal of minimizing the difference between the generated image and the reference image. The process is as follows: 1) Define a loss function to evaluate the difference between the generated image and the reference image. In this embodiment, the loss function used is the mean square error function (MSE). 2) Gradient descent optimization: Calculate the gradients of the two low-rank matrices A and B in the LoRa matrix based on the loss function (the gradient indicates how to adjust A and B to minimize the loss function), and use the gradient descent method or other optimization algorithms (such as the Adam optimizer) to update the weight parameters of the low-rank matrices A and B; 3) Repeat the above steps, i.e., calculate the gradient and update the parameters, until the value of the loss function reaches the preset threshold or the number of iterations reaches the preset maximum value.

[0025] In this embodiment, if Figure 2 As shown, according to the above method steps, the system includes: Pre-trained model loading module: used to load existing or trained pre-trained models; Dataset preparation module: responsible for selecting, preprocessing and annotating training images; LoRa matrix configuration module: used to define LoRa matrix parameters, instantiate LoRa matrix configuration objects, and complete the transfer of LoRa matrix parameters to the target layer of the pre-trained model; LoRa model training module: used to train the pre-trained model containing the LoRa matrix to obtain the target LoRa model; Image generation module: Use the trained LoRa model to generate an image similar to the target image based on the input conditions.

[0026] The present invention uses the LoRa model to generate similar images with the characteristics of high efficiency and flexibility. Only a small number of parameters in the pre-trained model (i.e., the LoRa matrix) need to be adjusted, which greatly reduces the demand for computing resources and training time. Users can adjust the parameters and input parameters of the LoRa matrix as needed to generate similar images with different styles and characteristics. At the same time, due to the introduction of the LoRa matrix, the model can generate high-quality similar images while maintaining the performance of the original pre-trained model; and can be easily integrated into various image generation systems.

[0027] It should be particularly noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0028] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a ROM, a RAM, etc.

[0029] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for generating similar images using a LoRa model, characterized in that: The following steps are involved: Step 1: Load the pre-trained model: Load the required pre-trained model based on the image generation task; Step 2: Configure the LoRa matrix: According to the importance of each layer in the pre-trained model and the requirements of the image generation task, clearly define the target layer where the LoRa matrix needs to be applied, and define the LoRa matrix parameters; Step 3: Apply the LoRa matrix to the pre-trained model: Instantiate the LoRa matrix configuration object, pass the parameters of the LoRa matrix to the target layer of the pre-trained model through the PEFT technology, and add the LoRa matrix to the target layer; and fix the rest of the pre-trained model, allowing only the weights of the LoRa matrix to be updated during the training process; Step 4: Prepare training data set: select high-quality images similar to reference images as training set and preprocess the training set; Step 5: Train the LoRa model: Use the preprocessed training data set to train the pre-trained model containing the LoRa matrix, and adjust the weight of the LoRa matrix through the optimization algorithm to minimize the difference between the generated image and the reference image to obtain the LoRa model; Step 6. Generate similar images: Generate images similar to the reference image through the LoRa model.

2. A method for generating similar images using a LoRa model according to claim 1, characterized in that: The pre-trained model is obtained after training with a large amount of data.

3. A method for generating similar images using a LoRa model according to claim 1, characterized in that: The LoRa matrix parameters include rank, scaling factor and dropout rate.

4. The method for generating similar images using the LoRa model according to claim 1, characterized in that: The preprocessing process in step 4 includes adjusting image size, cropping and normalization.

5. The method for generating similar images using the LoRa model according to claim 1, characterized in that: The specific process of adjusting the weight of the LoRa matrix by the optimization algorithm in step 5 is: Step 501, define a loss function: used to evaluate the difference between the generated image and the reference image; Step 502, gradient descent optimization: Calculate the gradients of two low-rank matrices A and B in the LoRa matrix based on the loss function, and use the gradient descent method or Adam optimizer to update the weight parameters of the low-rank matrices A and B; Step 503: Repeat steps 501 and 502 until the value of the loss function reaches a preset threshold or the number of iterations reaches a preset maximum value.

6. A method for generating similar images using a LoRa model according to claim 5, characterized in that: The loss function adopts a mean square error function.

7. The method for generating similar images using the LoRa model according to claim 1, characterized in that: The training process in step 5 also includes monitoring the loss value and the quality of the generated images during the training process to ensure that the model is trained as expected.

8. The method for generating similar images using the LoRa model according to claim 1, characterized in that: The instantiated LoRa matrix configuration object is specifically: setting the LoRa class in the PEFT library to specify the task type, the target layer to which the LoRa matrix needs to be applied, defining the LoRa matrix parameters, and passing in all configuration parameters.

9. A system for generating similar images using a LoRa model, applying a method for generating similar images using a LoRa model as described in any one of claims 1 to 8, characterized in that: include: Pre-trained model loading module: used to load existing or trained pre-trained models; Dataset preparation module: responsible for selecting, preprocessing and annotating training images; LoRa matrix configuration module: used to define LoRa matrix parameters, instantiate LoRa matrix configuration objects, and complete the transfer of LoRa matrix parameters to the target layer of the pre-trained model; LoRa model training module: used to train the pre-trained model containing the LoRa matrix to obtain the target LoRa model; Image generation module: Use the trained LoRa model to generate an image similar to the target image based on the input conditions.

Images