Fine-tuning method and device based on large model, equipment and readable storage medium

Abstract

The application discloses a fine-tuning method and device based on a large model, equipment and a readable storage medium, and the method comprises the following steps: acquiring a first data set and a base class large model used for detecting specific objects in the first data set, wherein each picture in the first data set contains specific objects and non-specific objects; acquiring a prompt word group used for representing a specific object category, inputting the prompt word group into the base class large model for identification, and obtaining a target category of the specific object; matching training fine-tuning parameters through the target category of the specific object, inserting the training fine-tuning parameters into a structure layer of the base class large model, and generating a to-be-trained model, wherein the training fine-tuning parameters comprise a low-rank matrix scaling factor, a rank of a low-rank matrix, an overfitting control factor, an inserted module and a layer name; acquiring a second data set containing only specific objects of the target category, and training the to-be-trained model through the second data set, so as to realize fine-tuning of the large model. The application reduces the occupation of computing resources.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a fine-tuning method, apparatus, device, and readable storage medium based on a large model. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-scale models have been widely used in various fields. Specifically, large-scale models refer to machine learning models with extremely large parameters (usually over a billion) and complex computational structures, capable of processing massive amounts of data and completing various complex tasks, such as natural language processing and image recognition. Large-scale models employ a pre-training + fine-tuning training model, and after training on large-scale data, they can quickly adapt to a series of downstream tasks.

[0003] Currently, there are several methods for fine-tuning large models:

[0004] 1) Full Fine-Tuning: Full fine-tuning is a method that updates all parameters of a pre-trained model. This method is suitable for situations with sufficient training data and computational resources, enabling the model to better adapt to a specific task. The disadvantages are that it requires significant computational resources and storage space, has a long training time, especially on large datasets, and carries a higher risk of overfitting when data is insufficient. Furthermore, running multiple fine-tuned models requires a significant increase in computational resources.

[0005] 2) LoRA Fine-tuning: LoRA fine-tuning refers to adjusting the model by introducing low-rank matrices into specific parts of the model without changing the original model weights. This method significantly reduces the number of parameters that need to be trained, thereby reducing the demand for computational resources while maintaining improved model performance. In this method, a fine-tuned model needs to be attached to a large base class model, and multiple fine-tuned large models for multiple specific tasks require multiple large base class models, which also requires a lot of computational resources. Moreover, the LoRA method requires modification of its network layer architecture when developing the multi-head attention mechanism.

[0006] 3) Adapter Fine-tuning: Adapter fine-tuning achieves fine-tuning by adding a lightweight adapter module to the model without affecting the original model structure. The adapter can be optimized for specific tasks, providing a flexible and computationally inexpensive fine-tuning method. This method shares the same drawbacks as the second method, while increasing model depth and training time. Furthermore, the design and adjustment of the adapter require considerable operational experience.

[0007] There is currently no effective solution to the problem that fine-tuning methods that update all parameters of a pre-trained model in existing technologies consume a lot of computational resources. Summary of the Invention

[0008] The purpose of this invention is to provide a fine-tuning method, apparatus, device, and readable storage medium based on a large model, which can solve the technical problem that fine-tuning methods that update all parameters of a pre-trained model result in high computational resource consumption.

[0009] One aspect of the present invention provides a fine-tuning method based on a large model, the method comprising: acquiring a first dataset and a base class large model for detecting specific objects in the first dataset, wherein each image in the first dataset contains specific objects and non-specific objects; acquiring a cue word group for representing the category of the specific object, inputting the cue word group into the base class large model for recognition to obtain the target category of the specific object; training fine-tuning parameters by matching the target category of the specific object, inserting the training fine-tuning parameters into the structural layer of the base class large model to generate a model to be trained, wherein the training fine-tuning parameters include a low-rank matrix scaling factor, the rank of the low-rank matrix, an overfitting control factor, the inserted module, and the layer name; acquiring a second dataset containing only specific objects of the target category, and training the model to be trained using the second dataset to achieve fine-tuning of the large model.

[0010] Optionally, the prompt word group is input into a base class large model for recognition to obtain the target category of the specific object, including: using each prompt word in the prompt word group as input data of the base class large model for image recognition to obtain all recognition objects corresponding to each prompt word in the first dataset; acquiring a preset standard object image set, traversing the preset object image set, and determining the standard object image corresponding to the prompt word group; performing similarity matching between the standard object image and all recognition objects corresponding to each prompt word, and taking the recognition objects with a similarity greater than or equal to a first preset threshold as the specific object of the prompt word; calculating the number of specific objects for each prompt word, and taking the prompt word with the most specific objects as the target category of the specific object.

[0011] Optionally, training fine-tuning parameters by matching the target category of a specific object includes: obtaining a preset model fine-tuning rule table, wherein the preset model fine-tuning rule table includes item names and fine-tuning parameters, with the item names corresponding to the fine-tuning parameters; performing text similarity matching between the target category of the specific object and the item name; if there is an item name with a text similarity greater than or equal to a second preset threshold, determining the item name as the training fine-tuning parameter of the specific object; if there is no item name with a text similarity greater than or equal to the second preset threshold, performing semantic similarity matching between the target category of the specific object and the item name; if there is an item name with a semantic similarity greater than or equal to a third preset threshold, determining the item name as the training fine-tuning parameter of the specific object; if there is no item name with a semantic similarity greater than or equal to the third preset threshold, obtaining manually set fine-tuning parameters as the training fine-tuning parameters of the specific object, and storing the target category of the specific object and the manually set fine-tuning parameters in the preset model fine-tuning rule table.

[0012] Optionally, the training fine-tuning parameters include multiple fine-tuning parameter sets. The training model is trained using the second dataset to achieve fine-tuning of the large model. This includes: freezing the structural parameters of the base class large model and selecting a fine-tuning parameter set from the training fine-tuning parameters as a fine-tuning layer to generate multiple first training models; inputting the second dataset into the first training models for training, performing forward propagation calculations through each layer of the model, and outputting the predicted category of a specific object in the second dataset; calculating the loss function value between the predicted category of the first training model corresponding to each fine-tuning parameter set and the true label of the second dataset; determining the target fine-tuning parameter set based on the loss function value, and constructing a fine-tuned large model using the fine-tuning layer composed of the target fine-tuning parameter set and the base class large model.

[0013] Optionally, the target fine-tuning parameter set is determined by the loss function value, including: comparing the magnitudes of different loss function values, determining the fine-tuning parameter set corresponding to the minimum loss function value as the target fine-tuning parameter set; and saving the target fine-tuning parameter set to the database in the form of a model weight file.

[0014] Optionally, after obtaining a second dataset containing only specific objects of the target category and training the model to be trained using the second dataset to fine-tune the large model, the method further includes: in response to a user's detection operation on multiple specific objects, obtaining prompt words for multiple specific objects; obtaining a preset inference configuration table, wherein the preset inference configuration table includes prompt words, model weight file name, model weight file path, large model output result filtering threshold, and label number; determining the model weight file name, model weight file path, large model output result filtering threshold, and label number for each specific object in the preset inference configuration table based on the prompt words for multiple specific objects; determining the target model weight file based on the model weight file name and / or model weight file path; and loading the target model weight file and the large model output result filtering threshold into the base class large model according to the label number to achieve parallel inference for multiple specific objects.

[0015] Another aspect of the present invention provides a fine-tuning apparatus based on a large model, the apparatus comprising: an acquisition module for acquiring a first dataset and a base class large model for detecting specific objects in the first dataset, wherein each image in the first dataset contains specific objects and non-specific objects; a recognition module for acquiring cue words representing the category of the specific objects, inputting the cue words into the base class large model for recognition to obtain the target category of the specific objects; a fine-tuning module for training fine-tuning parameters by matching the target category of the specific objects, inserting the trained fine-tuning parameters into the structural layers of the base class large model to generate a model to be trained, wherein the trained fine-tuning parameters include a low-rank matrix scaling factor, the rank of the low-rank matrix, an overfitting control factor, the inserted module, and the layer name; and a training module for acquiring a second dataset containing only specific objects of the target category, and training the model to be trained using the second dataset to achieve fine-tuning of the large model.

[0016] Optionally, the device further includes a parallel inference module, configured to: in response to a user's detection operation on multiple specific objects, acquire prompt words for multiple specific objects; acquire a preset inference configuration table, wherein the preset inference configuration table includes prompt words, model weight file name, model weight file path, large model output result filtering threshold, and tag number; determine the model weight file name, model weight file path, large model output result filtering threshold, and tag number for each specific object in the preset inference configuration table based on the prompt words for multiple specific objects; determine the target model weight file based on the model weight file name and / or model weight file path; and load the target model weight file and the large model output result filtering threshold into the base class large model according to the tag number, so as to achieve parallel inference for multiple specific objects.

[0017] Another aspect of the present invention provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the fine-tuning method based on a large model of any of the above embodiments.

[0018] Another aspect of the present invention provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the fine-tuning method based on a large model of any of the above embodiments. Further, the computer-readable storage medium may primarily include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, etc.; and the data storage area may store data created based on the use of blockchain nodes, etc.

[0019] This invention determines the target category of a feature object by using prompt words for the feature object, and further matches the corresponding training fine-tuning parameters based on the target category. The fine-tuning parameters are only inserted into the base class model for data adjustment during the training process, which reduces the computational resource consumption and data volume of the base class model in the training and inference process of specific object category detection, thus saving computational resources. Attached Figure Description

[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0021] Figure 1 A flowchart of an optional fine-tuning method based on a large model provided in Embodiment 1 of the present invention is shown;

[0022] Figure 2 A structural block diagram of the fine-tuning device based on a large model provided in Embodiment 2 of the present invention is shown; and

[0023] Figure 3 A block diagram of a computer device suitable for implementing a large-model-based fine-tuning method is shown in Embodiment 3 of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0025] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0026] Example 1

[0027] This embodiment provides a fine-tuning method based on a large model. Figure 1 The flowchart of this large-model-based fine-tuning method is shown, as follows: Figure 1 As shown, this fine-tuning method based on a large model may include steps S101 to S104, wherein:

[0028] Step S101: Obtain the first dataset and the base class large model for detecting specific objects in the first dataset, wherein each image in the first dataset contains specific objects and non-specific objects;

[0029] The first dataset is the base image set obtained by the user to perform a specific object detection task. The images in the first dataset must contain at least the specific object to be detected, and may also contain information unrelated to the specific object. There is no limit to the number of images in the first dataset.

[0030] The base class model can be a transform neural network model or other network models with similar execution principles; there are no restrictions here.

[0031] Step S102: Obtain the prompt words used to represent the category of a specific object, input the prompt words into the base class large model for recognition, and obtain the target category of the specific object;

[0032] The prompt phrases can include names describing a specific object category, synonyms / near-synonyms of the name, foreign language explanations of the name, and user-defined words. For example, if a user needs to detect chairs in a set of images, the prompt phrases they need to obtain could be "chair", "stool", or "chair".

[0033] Input a prompt for a specific object into the system. Use a base class model to detect the target in the test image based on the prompt. Obtain the baseline detection performance of the base class model for the specific target under each prompt condition. Select the prompt with the best detection performance from among the prompts as the category name for this type of target.

[0034] Step S103: Train fine-tuning parameters by matching the target category of a specific object, insert the training fine-tuning parameters into the structural layer of the base class large model, and generate the model to be trained. The training fine-tuning parameters include the low-rank matrix scaling factor, the rank of the low-rank matrix, the overfitting control factor, the inserted module and the layer name.

[0035] Pre-trained base class models often encompass massive amounts of data, frequently leading to insufficient storage and computational resources during application. This embodiment trains fine-tuned parameters through target category matching of specific objects, then inserts these fine-tuned parameters into the structural layers of the base class model for training. This achieves the same effectiveness as LoRA layers without modifying the internal structure of the multi-head attention layer, saving computational and storage resources during training. Fine-tuning techniques allow these models to adapt to specific application scenarios or tasks, improving their performance in those scenarios and the detection accuracy of the base class model for specific objects.

[0036] Before inserting the trained fine-tuning parameters into the structural layers of the base class large model, the environment required for the fine-tuning task is initialized, including data loading, accumulating large model structure generation, fine-tuning layer initialization and insertion, etc. After initialization, the fine-tuning layers are inserted into the base class large model, the parameters of the base class large model are frozen, and the parameters of the fine-tuning layers are set to a trainable state. These trainable fine-tuning layers are called fine-tuning adapters.

[0037] Step S104: Obtain a second dataset containing only specific objects of the target category, and train the model to be trained using the second dataset to achieve fine-tuning of the large model.

[0038] Since the target category is the data with the best prediction performance of the base class large model, using this as a benchmark to collect images containing only specific objects of the target category can enhance the detection performance of the model to be trained.

[0039] Specifically, 200 images can be collected, and the training and test sets can be divided in a 4:1 or 5:1 ratio. The collected images are labeled using the COCO dataset format. The labeling information includes bounding box information and category information for specific objects. The bounding box information includes the absolute x-coordinate and y-coordinate of the top-left corner, the absolute width of the bounding box, and the absolute height of the bounding box.

[0040] Preferably, step S102 may include steps S1021 to S1024, wherein:

[0041] Step S1021: Use each prompt word in the prompt word group as input data for the base class large model to perform image recognition, and obtain all the recognition objects corresponding to each prompt word in the first dataset;

[0042] Step S1022: Obtain a preset standard object image set, traverse the preset object image set, and determine the standard object image corresponding to the prompt phrase;

[0043] A standard object image set can be a historical image set containing only specific objects. The standard object image set contains various types of standard object images, and the type name can be defined according to the object name, such as chair, fire extinguisher, cup, etc.

[0044] Step S1023: Perform similarity matching between the standard object image and all recognized objects corresponding to each prompt word, and take the recognized objects with similarity greater than or equal to the first preset threshold as the specific objects of the prompt word;

[0045] The first preset threshold is determined by the needs of the scenario and is not limited here.

[0046] Step S1024: Calculate the number of specific objects for each prompt word, and take the prompt word with the most specific objects as the target category of the specific objects.

[0047] By using prompt words as input data for the base class model, the detection effect of the same specific object in different categories is realized based on the base class model. The prompt word with the best detection effect (the largest number of specific objects detected) is selected from multiple prompt words as the category name of this type of target, which facilitates the accurate definition of specific object detection by users in the future, thereby improving the recognition accuracy of the base class model.

[0048] The base class large model is used to perform benchmark testing on the test images to obtain the benchmark detection performance of the base class large model for a specific target.

[0049] Preferably, step S103 may include steps S1031 to S1034, wherein:

[0050] Step S1031: Obtain the preset model fine-tuning rule table, wherein the preset model fine-tuning rule table includes project name and fine-tuning parameters, and the project name corresponds to the fine-tuning parameters;

[0051] The project name is the standard category name of the specific object.

[0052] Step S1032: Perform text similarity matching between the target category and the project name of the specific object. If there is a project name with a text similarity greater than or equal to the second preset threshold, determine the project name as the training fine-tuning parameter of the specific object.

[0053] Step S1033: If there is no item name with text similarity greater than or equal to the second preset threshold, perform semantic similarity matching between the target category and the item name of the specific object; if there is an item name with semantic similarity greater than or equal to the third preset threshold, determine the item name as the training fine-tuning parameter of the specific object.

[0054] Step S1034: If there is no item name with semantic similarity greater than or equal to the third preset threshold, obtain the manually set fine-tuning parameters as the training fine-tuning parameters for the specific object, and store the target category of the specific object and the manually set fine-tuning parameters in the preset model fine-tuning rule table.

[0055] Since the obtained prompts lack a standardized description format, directly using the target category of a specific object as the search unit to traverse the preset model fine-tuning rule table may result in a "no search object" situation, making it impossible to fine-tune the base class model. This embodiment aims to determine the fine-tuning parameters that match the target category of a specific object with the item name through text similarity and semantic similarity matching. Furthermore, if the fine-tuning parameters cannot be determined after the aforementioned two methods, manually set fine-tuning parameters are used as the training fine-tuning parameters for the specific object, providing data support for the subsequent training process of the base class model and reducing the computational resources consumed during training.

[0056] Preferably, the training fine-tuning parameters include multiple fine-tuning parameter sets, and step S104 may include steps S1041 to S1044, wherein:

[0057] Step S1041: Freeze the structural parameters of the base class large model, and select a fine-tuning parameter set from the training fine-tuning parameters as a fine-tuning layer to generate multiple first models to be trained;

[0058] Step S1042: Input the second dataset into the first model to be trained for training. Perform forward propagation calculation through each layer of the model to output the predicted category of the specific object in the second dataset.

[0059] Step S1043: Calculate the loss function value between the predicted category of the first model to be trained and the true label of the second dataset for each fine-tuning parameter set;

[0060] Step S1044: Determine the target fine-tuning parameter set through the loss function value, and construct the fine-tuned large model using the fine-tuning layer composed of the target fine-tuning parameter set and the base class large model.

[0061] Specifically, the target fine-tuning parameter set is determined through the loss function value, including steps A1 to A2, wherein:

[0062] Step A1: Compare the values ​​of different loss functions and determine the set of fine-tuning parameters corresponding to the minimum loss function value as the target fine-tuning parameter set;

[0063] Step A2: Save the target fine-tuning parameter set to the database as a model weight file.

[0064] After full fine-tuning, the output model weights of a large base class model can reach hundreds of MB or even GB, but the model weight file is only a few MB in size, which can greatly save storage resources.

[0065] In addition, training begins on the base class large model with the fine-tuning layer inserted. During this process, users can select appropriate training parameters (such as learning rate, number of training iterations, number of training images in the input batch, etc.) to fine-tune the training of the new large model.

[0066] Preferably, after obtaining a second dataset containing only specific objects of the target category and training the model to be trained using the second dataset to fine-tune the large model, steps B1 to B5 are further included, wherein:

[0067] Step B1: In response to the user's detection operation on multiple specific objects, obtain prompt words for multiple specific objects;

[0068] Step B2: Obtain the preset inference configuration table, which includes prompt words, model weight file name, model weight file path, large model output result filtering threshold, and label number.

[0069] There is a unique mapping relationship between the prompt words, the model weight file name, the model weight file path, the large model output filtering threshold, and the label number.

[0070] Step B3: Determine the model weight file name, model weight file path, large model output result filtering threshold, and label number for each specific object in the preset inference configuration table using prompt words for multiple specific objects;

[0071] Step B4: Determine the target model weight file by the model weight file name and / or model weight file path;

[0072] Step B5: Load the target model weight file and the large model output filtering threshold into the base class large model according to the label sequence number to achieve parallel inference for multiple specific objects.

[0073] In real-world business scenarios, users may need to perform inference tasks involving multiple specific objects. Traditional fine-tuning inference methods, where one fine-tuning adapter plus a large base model, is only effective for one specific task, require multiple fine-tuning adapters plus multiple large base models to be applied sequentially for multiple tasks. This approach consumes significant computational resources and is detrimental to the efficient execution of the training process. This embodiment addresses this by inserting multiple fine-tuning adapters into a single large base model, enabling parallel inference using multiple fine-tuning models. With minimal increase in GPU memory resources, it achieves results that are effective for multiple specific tasks in a single inference operation.

[0074] The following is an example of actual user operation:

[0075] The system deploys the model by constructing a large inference model based on the number of detection tasks input by the user, and then infers the data sent in the task scenario. Users can set the following parameters to construct the large inference model:

[0076] a.num_adapters: The number of fine-tuning adapters loaded.

[0077] b.name: The name of each fine-tuning adapter

[0078] c.model_path: The weight path for each fine-tuning adapter model.

[0079] d.prompt: A prompt word for each specific task objective.

[0080] e.threshold: The filtering threshold for the output of each large model.

[0081] f.label_num: The label number used for each target.

[0082] After the model deployment begins processing user data, the results of model inference that have issues with user feedback can be collected and added to the training dataset for fine-tuning adapter training iterations for specific tasks.

[0083] This embodiment determines the target category of a feature object by using prompt words for the feature object, and further matches the corresponding training fine-tuning parameters based on the target category. The fine-tuning parameters are only inserted into the base class model for data adjustment during the training process, which reduces the computational resource consumption and data volume of the base class model in the training and inference process of specific object category detection, thus saving computational resources.

[0084] Example 2

[0085] Embodiment 2 of the present invention also provides a fine-tuning device based on a large model. This fine-tuning device corresponds to the fine-tuning method based on a large model provided in Embodiment 1 above. The corresponding technical features and effects will not be detailed in this embodiment, but relevant aspects can be referred to Embodiment 1 above. Specifically, Figure 2 A structural block diagram of this fine-tuning device based on a large model is shown. Figure 2 As shown, the large model-based fine-tuning device 200 includes an acquisition module 201, an identification module 202, a fine-tuning module 203, and a training module 204, wherein:

[0086] The acquisition module 201 is used to acquire a first dataset and a base class large model for detecting specific objects in the first dataset, wherein each image in the first dataset contains specific objects and non-specific objects;

[0087] The recognition module 202, connected to the acquisition module 201, is used to acquire a prompt word group that represents the category of a specific object, input the prompt word group into the base class large model for recognition, and obtain the target category of the specific object;

[0088] The fine-tuning module 203, connected to the recognition module 202, is used to train fine-tuning parameters by matching the target category of a specific object, insert the trained fine-tuning parameters into the structural layer of the base class large model, and generate the model to be trained. The trained fine-tuning parameters include the low-rank matrix scaling factor, the rank of the low-rank matrix, the overfitting control factor, the inserted module and the layer name.

[0089] The training module 204, connected to the fine-tuning module 203, is used to obtain a second dataset containing only specific objects of the target category, and to train the model to be trained using the second dataset in order to fine-tune the large model.

[0090] Optionally, the recognition module is specifically used for: using each prompt word in the prompt word group as input data for the base class large model to perform image recognition, thereby obtaining all recognition objects corresponding to each prompt word in the first dataset; acquiring a preset standard object image set, traversing the preset object image set, and determining the standard object image corresponding to the prompt word group; performing similarity matching between the standard object image and all recognition objects corresponding to each prompt word, and taking the recognition objects with a similarity greater than or equal to a first preset threshold as the specific objects of the prompt word; calculating the number of specific objects for each prompt word, and taking the prompt word with the most specific objects as the target category of the specific objects.

[0091] Optionally, the fine-tuning module is specifically used for: obtaining a preset model fine-tuning rule table, wherein the preset model fine-tuning rule table includes project names and fine-tuning parameters, with project names corresponding to fine-tuning parameters; performing text similarity matching between the target category and project names of a specific object; if there is a project name with a text similarity greater than or equal to a second preset threshold, determining the project name as the training fine-tuning parameter for the specific object; if there is no project name with a text similarity greater than or equal to the second preset threshold, performing semantic similarity matching between the target category and project names of the specific object; if there is a project name with a semantic similarity greater than or equal to a third preset threshold, determining the project name as the training fine-tuning parameter for the specific object; if there is no project name with a semantic similarity greater than or equal to the third preset threshold, obtaining manually set fine-tuning parameters as the training fine-tuning parameters for the specific object, and storing the target category and manually set fine-tuning parameters of the specific object in the preset model fine-tuning rule table.

[0092] Optionally, the training module is also used to: freeze the structural parameters of the base class large model, and select a fine-tuning parameter set from the training fine-tuning parameters as a fine-tuning layer to generate multiple first models to be trained; input the second dataset into the first models to be trained for training, perform forward propagation calculations through each layer of the model, and output the predicted category of a specific object in the second dataset; calculate the loss function value between the predicted category of the first model to be trained corresponding to each fine-tuning parameter set and the true label of the second dataset; determine the target fine-tuning parameter set through the loss function value, and construct a fine-tuned large model with the fine-tuning layer composed of the target fine-tuning parameter set and the base class large model.

[0093] Optionally, the training module is also used to: compare the magnitudes of different loss function values, determine the fine-tuning parameter set corresponding to the minimum loss function value as the target fine-tuning parameter set, and save the target fine-tuning parameter set to the database in the form of a model weight file.

[0094] Optionally, the device further includes a parallel inference module, configured to: in response to a user's detection operation on multiple specific objects, acquire prompt words for multiple specific objects; acquire a preset inference configuration table, wherein the preset inference configuration table includes prompt words, model weight file name, model weight file path, large model output result filtering threshold, and tag number; determine the model weight file name, model weight file path, large model output result filtering threshold, and tag number for each specific object in the preset inference configuration table based on the prompt words for multiple specific objects; determine the target model weight file based on the model weight file name and / or model weight file path; and load the target model weight file and the large model output result filtering threshold into the base class large model according to the tag number, so as to achieve parallel inference for multiple specific objects.

[0095] Example 3

[0096] Figure 3 A block diagram of a computer device suitable for implementing a large-model-based fine-tuning method is shown in Embodiment 3 of the present invention. In this embodiment, the computer device 300 may be a smartphone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers), etc., that executes a program. Figure 3 As shown, the computer device 300 in this embodiment includes, but is not limited to, a memory 301, a processor 302, and a network interface 303 that are communicatively connected to each other via a system bus. It should be noted that... Figure 3 Only a computer device 300 with components 301-303 is shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0097] In this embodiment, the memory 303 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 301 may be an internal storage unit of the computer device 300, such as the hard disk or memory of the computer device 300. In other embodiments, the memory 301 may also be an external storage device of the computer device 300, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 300. Of course, the memory 301 may also include both the internal storage unit and the external storage device of the computer device 300. In this embodiment, the memory 301 is typically used to store the operating system and various application software installed on the computer device 300, such as program code for fine-tuning methods based on large models.

[0098] In some embodiments, processor 302 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 302 is typically used to control the overall operation of computer device 300. For example, it performs control and processing related to data interaction or communication with computer device 300. In this embodiment, processor 302 is used to run program code stored in memory 301 for the steps of a fine-tuning method based on a large model.

[0099] In this embodiment, the fine-tuning method based on a large model stored in memory 301 can also be divided into one or more program modules and executed by one or more processors (processor 302 in this embodiment) to complete the present invention.

[0100] Network interface 303 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 300 and other computer devices. For example, network interface 303 is used to connect computer device 300 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 300 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0101] Example 4

[0102] This embodiment also provides a computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., which stores a computer program. When the computer program is executed by a processor, it implements the steps of a fine-tuning method based on a large model.

[0103] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0104] It should be noted that the sequence numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0106] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A fine-tuning method based on a large model, characterized in that, The method includes: Obtain a first dataset and a large base class model for detecting specific objects in the first dataset, wherein each image in the first dataset contains specific and non-specific objects; Obtain a set of prompt words to represent a specific object category, input the prompt words into the base class model for recognition, and obtain the target category of the specific object; The training fine-tuning parameters are trained by matching the target category of the specific object, and then inserted into the structural layer of the base class large model to generate a model to be trained. The training fine-tuning parameters include the low-rank matrix scaling factor, the rank of the low-rank matrix, the overfitting control factor, the inserted module and the layer name. Obtain a second dataset containing only specific objects of the target category, and train the model to be trained using the second dataset to achieve fine-tuning of the large model; The method further includes: In response to the user's detection operation on multiple specific objects, obtain prompts for multiple specific objects; Obtain a preset inference configuration table, wherein the preset inference configuration table includes prompt words, model weight file name, model weight file path, large model output result filtering threshold and label number; The model weight file name, model weight file path, large model output result filtering threshold, and label number of each specific object are determined by the prompt words of multiple specific objects in the preset inference configuration table; The target model weight file is determined by the model weight file name and / or the model weight file path; The target model weight file and the large model output filtering threshold are loaded into the base class large model according to the label sequence number to achieve parallel inference for multiple specific objects.

2. The method according to claim 1, characterized in that, The step of inputting the prompt phrase into the base class model for recognition to obtain the target category of the specific object includes: Each prompt word in the prompt word group is used as input data for image recognition in the base class large model to obtain all recognition objects corresponding to each prompt word in the first dataset; Obtain a preset set of standard object images, traverse the preset set of standard object images, and determine the standard object image corresponding to the prompt phrase; The standard object image is matched with all the recognition objects corresponding to each prompt word, and the recognition objects with a similarity greater than or equal to a first preset threshold are identified as the specific objects of the prompt word; Calculate the number of specific objects for each prompt word, and take the prompt word with the most specific objects as the target category of the specific objects.

3. The method according to claim 1 or 2, characterized in that, The fine-tuning of parameters through target category matching of the specific object includes: Obtain a preset model fine-tuning rule table, wherein the preset model fine-tuning rule table includes project names and fine-tuning parameters, and the project names correspond to the fine-tuning parameters; The target category of the specific object is matched with the project name using text similarity. If there is a project name with a text similarity greater than or equal to a second preset threshold, the project name is determined as a training fine-tuning parameter for the specific object. If there is no item name with a text similarity greater than or equal to the second preset threshold, perform semantic similarity matching between the target category of the specific object and the item name. If there is an item name with a semantic similarity greater than or equal to the third preset threshold, determine the item name as the training fine-tuning parameter of the specific object. If there is no project name with semantic similarity greater than or equal to the third preset threshold, the manually set fine-tuning parameters are obtained as the training fine-tuning parameters of the specific object, and the target category of the specific object and the manually set fine-tuning parameters are stored in the preset model fine-tuning rule table.

4. The method according to claim 1, characterized in that, The training fine-tuning parameters include multiple sets of fine-tuning parameters. The step of training the model to be trained using the second dataset to achieve fine-tuning of a large model includes: Freeze the structural parameters of the base class model, and select a fine-tuning parameter set from the training fine-tuning parameters as a fine-tuning layer to generate multiple first models to be trained. The second dataset is fed into the first model to be trained. Forward propagation is performed through each layer of the model to output the predicted category of a specific object in the second dataset. Calculate the loss function value between the predicted category of the first model to be trained and the true label of the second dataset for each fine-tuning parameter set; The target fine-tuning parameter set is determined by the loss function value, and a fine-tuned large model is constructed using the fine-tuning layer composed of the target fine-tuning parameter set and the base class large model.

5. The method according to claim 4, characterized in that, The step of determining the target fine-tuning parameter set through the loss function value includes: By comparing the values ​​of different loss functions, the set of fine-tuning parameters corresponding to the minimum loss function value is determined as the target fine-tuning parameter set; The target fine-tuning parameter set is saved to the database as a model weight file.

6. A fine-tuning device based on a large model, characterized in that, The device includes: The acquisition module is used to acquire a first dataset and a base class large model for detecting specific objects in the first dataset, wherein each image in the first dataset contains specific objects and non-specific objects; The recognition module is used to obtain a prompt word group to represent the category of a specific object, input the prompt word group into the base class model for recognition, and obtain the target category of the specific object; The fine-tuning module is used to train fine-tuning parameters by matching the target category of the specific object, and insert the trained fine-tuning parameters into the structural layer of the base class large model to generate a model to be trained. The trained fine-tuning parameters include a low-rank matrix scaling factor, the rank of the low-rank matrix, an overfitting control factor, the inserted module and the layer name. The training module is used to obtain a second dataset containing only specific objects of the target category, and to train the model to be trained using the second dataset in order to fine-tune the large model. The device further includes a parallel inference module, used for: In response to the user's detection operation on multiple specific objects, obtain prompts for multiple specific objects; Obtain a preset inference configuration table, wherein the preset inference configuration table includes prompt words, model weight file name, model weight file path, large model output result filtering threshold and label number; The model weight file name, model weight file path, large model output result filtering threshold, and label number of each specific object are determined by the prompt words of multiple specific objects in the preset inference configuration table; The target model weight file is determined by the model weight file name and / or the model weight file path; The target model weight file and the large model output filtering threshold are loaded into the base class large model according to the label sequence number to achieve parallel inference for multiple specific objects.

7. A computer device, the computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 5.

Images