Fine adjustment method, device and equipment based on large model and readable storage medium

Through the prompt phrase matching of feature objects, training fine-tuning parameters are only inserted into the base class large model for fine-tuning, which solves the problem of high consumption of full fine-tuning computing resources, and achieves resource saving and model adaptability improvement.

CN119006981AActive Publication Date: 2024-11-22GLODON CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411210296.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-22
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

In the existing large-scale fine-tuning methods, full fine-tuning leads to high consumption of computing resources, especially when the data volume is insufficient, the risk of overfitting is high. Although LoRA and Adapter fine-tuning reduce the demand for computing resources, a large amount of calculations still need to be made during multi-task fine-tuning. resources and increase model depth and training time.

Method used

By obtaining the prompt phrases of a specific object, matching the training fine-tuning parameters, inserting only the fine-tuning parameters into the structural layer of the base class large model, generating the model to be trained, and fine-tuning is performed through the data set of the target category, freezing the structural parameters of the base class large model. , reduce the use of computing resources.

Benefits of technology

It effectively reduces the computing resource occupation of large models in the detection and inference process of specific object categories, saves computing resources and data volume, and improves the adaptability and detection accuracy of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006981A_ABST
    Figure CN119006981A_ABST
Patent Text Reader

Abstract

The invention discloses a fine tuning method, device and equipment based on a large model and a readable storage medium, and the method comprises the steps: obtaining a first data set and a base class large model used for detecting a specific object of the first data set, and each picture in the first data set comprises the specific object and a non-specific object; obtaining a prompt word group used for representing the category of the specific object, and inputting the prompt word group into the base class large model for recognition to obtain a target category of the specific object; training fine tuning parameters are matched through a target category of a specific object, the training fine tuning parameters are inserted into a structural layer of the base class large model, a to-be-trained model is generated, and the training fine tuning parameters comprise a low-rank matrix scaling factor, a rank of a low-rank matrix, an overfitting control factor, an inserted module and a layer name; and obtaining a second data set only containing the specific object of the target category, and training the to-be-trained model through the second data set to realize fine tuning of the large model. According to the invention, computing resource occupation is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a fine-tuning method, device, equipment and readable storage medium based on a large model. Background Art

[0002] With the rapid development of artificial intelligence technology, large models have been widely used in various fields. Specifically, large models refer to machine learning models with ultra-large-scale parameters (usually more than one billion) and complex computing structures, which can process massive amounts of data and complete various complex tasks, such as natural language processing and image recognition. Large models use a pre-training + fine-tuning training mode. After training on large-scale data, they can quickly adapt to a series of downstream tasks.

[0003] There are several methods for fine-tuning large models:

[0004] 1) Full fine-tuning: Full parameter fine-tuning is a fine-tuning method that updates all parameters of a pre-trained model. This method is suitable for situations where there is sufficient training data and computing resources, and can make the model better suited to specific tasks. The disadvantage is that it requires a lot of computing resources and storage space, and the training time is long, especially on large data sets. There is a higher risk of overfitting when the amount of data is insufficient. In addition, the computing resources required to run multiple fine-tuned models increase exponentially.

[0005] 2) LoRA fine-tuning: LoRA fine-tuning refers to fine-tuning the model by introducing a low-rank matrix in a specific part of the model without changing the original model weights. This method significantly reduces the number of parameters that need to be trained, thereby reducing the demand for computing resources while maintaining the improvement of model performance. In this method, a fine-tuning model needs to be attached to a base class large model. Implementing multiple fine-tuning large models for multiple specific tasks requires multiple base class large models, which also requires a lot of computing resources. In addition, when the LoRA method sorts out the multi-head attention mechanism, its network layer architecture must be modified.

[0006] 3) Adapter fine-tuning: Adapter fine-tuning achieves fine-tuning by adding a lightweight adapter module to the model without affecting the original model structure. The adapter can be optimized for specific tasks, providing a flexible and low-cost fine-tuning method. This method has the same disadvantages as the second method, while increasing the model depth and lengthening the training time. The design and adjustment of the adapter require personnel experience.

[0007] There is currently no effective solution to the technical problem that the fine-tuning method in the prior art that updates all parameters of the pre-trained model leads to high consumption of computing resources. Summary of the invention

[0008] The purpose of the present invention is to provide a fine-tuning method, device, equipment and readable storage medium based on a large model, which can solve the technical problem that the fine-tuning method of updating all parameters of the pre-trained model leads to large consumption of computing resources.

[0009] One aspect of the present invention provides a fine-tuning method based on a large model, the method comprising: obtaining a first data set and a base class large model for detecting specific objects in the first data set, wherein each image in the first data set contains specific objects and non-specific objects; obtaining a prompt phrase for characterizing the category of a specific object, inputting the prompt phrase into the base class large model for identification, and obtaining a target category of the specific object; training fine-tuning parameters through target category matching of the specific object, inserting the training fine-tuning parameters into the structural layer of the base class large model, and generating a model to be trained, wherein the training fine-tuning parameters include a low-rank matrix scaling factor, a rank of the low-rank matrix, an overfitting control factor, an inserted module, and a layer name; obtaining a second data set containing only specific objects of the target category, and training the model to be trained through the second data set to achieve fine-tuning of the large model.

[0010] Optionally, the prompt phrase is input into the base class large model for recognition to obtain the target category of the specific object, including: using each prompt word in the prompt phrase as input data of the base class large model for image recognition to obtain all the recognized objects corresponding to each prompt word in the first data set; obtaining a preset standard object image set, traversing the preset object image set, and determining the standard object image corresponding to the prompt phrase; performing similarity matching between the standard object image and all the recognized objects corresponding to each prompt word, and taking the recognized objects with a similarity greater than or equal to a first preset threshold as the specific objects of the prompt word; calculating the number of specific objects for each prompt word, and taking the prompt word with the largest number of specific objects as the target category of the specific object.

[0011] Optionally, training fine-tuning parameters are matched by target category of a specific object, including: obtaining a preset model fine-tuning rule table, wherein the preset model fine-tuning rule table includes a project name and fine-tuning parameters, and the project name corresponds to the fine-tuning parameters; performing text similarity matching on the target category of the specific object and the project name, and if there is a project name with a text similarity greater than or equal to a second preset threshold, determining that the project name is the training fine-tuning parameter of the specific object; if there is no project name with a text similarity greater than or equal to the second preset threshold, performing semantic similarity matching on the target category of the specific object and the project name, and if there is a project name with a semantic similarity greater than or equal to a third preset threshold, determining that the project name is the training fine-tuning parameter of the specific object; if there is no project name with a semantic similarity greater than or equal to the third preset threshold, obtaining manually set fine-tuning parameters as the training fine-tuning parameters of the specific object, and storing the target category of the specific object and the manually set fine-tuning parameters in the preset model fine-tuning rule table.

[0012] Optionally, the training fine-tuning parameters include multiple fine-tuning parameter sets, and the model to be trained is trained through the second data set to achieve fine-tuning of the large model, including: freezing the structural parameters of the base class large model, and selecting a fine-tuning parameter set from the training fine-tuning parameters as the fine-tuning layer to generate multiple first models to be trained; inputting the second data sets into the first models to be trained for training, performing forward propagation calculations through each layer of the model, and outputting predicted categories of specific objects of the second data set; calculating the loss function value between the predicted category of the first model to be trained corresponding to each fine-tuning parameter set and the true label of the second data set; determining the target fine-tuning parameter set through the loss function value, and constructing a fine-tuned large model with the fine-tuning layer composed of the target fine-tuning parameter set and the base class large model.

[0013] Optionally, the target fine-tuning parameter set is determined by the loss function value, including: comparing the sizes of different loss function values, determining the fine-tuning parameter set corresponding to the minimum loss function value as the target fine-tuning parameter set; and saving the target fine-tuning parameter set in the form of a model weight file in a database.

[0014] Optionally, after obtaining a second data set containing only specific objects of the target category and training the to-be-trained model through the second data set to achieve fine-tuning of the large model, the method further includes: obtaining prompt words for multiple specific objects in response to a user's detection operation on multiple specific objects; obtaining a preset inference configuration table, wherein the preset inference configuration table includes prompt words, model weight file name, model weight file path, large model output result filtering threshold, and label serial number; determining the model weight file name, model weight file path, large model output result filtering threshold, and label serial number of each specific object in the preset inference configuration table through the prompt words of multiple specific objects; determining the target model weight file through the model weight file name and / or model weight file path; and loading the target model weight file and the large model output result filtering threshold into the base class large model according to the label serial number to achieve parallel inference of multiple specific objects.

[0015] Another aspect of the present invention provides a fine-tuning device based on a large model, which includes: an acquisition module, used to acquire a first data set and a base class large model for detecting specific objects in the first data set, wherein each image in the first data set contains specific objects and non-specific objects; an identification module, used to acquire a prompt phrase for characterizing a specific object category, input the prompt phrase into the base class large model for identification, and obtain a target category of the specific object; a fine-tuning module, used to match training fine-tuning parameters through the target category of the specific object, insert the training fine-tuning parameters into the structural layer of the base class large model, and generate a model to be trained, wherein the training fine-tuning parameters include a low-rank matrix scaling factor, a rank of the low-rank matrix, an overfitting control factor, an inserted module and a layer name; a training module, used to acquire a second data set containing only specific objects of the target category, and train the model to be trained through the second data set to achieve fine-tuning of the large model.

[0016] Optionally, the device also includes a parallel reasoning module, which is used to: obtain prompt words for multiple specific objects in response to the user's detection operation on multiple specific objects; obtain a preset reasoning configuration table, wherein the preset reasoning configuration table includes prompt words, model weight file name, model weight file path, large model output result filtering threshold and label serial number; determine the model weight file name, model weight file path, large model output result filtering threshold and label serial number of each specific object in the preset reasoning configuration table through the prompt words of multiple specific objects; determine the target model weight file through the model weight file name and / or model weight file path; load the target model weight file and the large model output result filtering threshold into the base class large model according to the label serial number to realize parallel reasoning for multiple specific objects.

[0017] Yet another aspect of the present invention provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the large model-based fine-tuning method of any of the above-mentioned embodiments when executing the computer program.

[0018] Another aspect of the present invention provides a computer storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the fine-tuning method based on a large model of any of the above embodiments is implemented. Further, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of a blockchain node, etc.

[0019] The present invention determines the target category of the feature object through the prompt phrase of the feature object, further matches the corresponding training fine-tuning parameters according to the target category, and only inserts the fine-tuning parameters into the base class large model for data adjustment during the training process, thereby reducing the computing resource occupancy and data volume of the base class large model in the training and reasoning process of specific object category detection, thereby saving computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0021] Figure 1 An optional flow chart of a fine-tuning method based on a large model provided in the first embodiment of the present invention is shown;

[0022] Figure 2 A structural block diagram of a fine-tuning device based on a large model provided in the second embodiment of the present invention is shown; and

[0023] Figure 3 A block diagram of a computer device suitable for implementing a large model-based fine-tuning method provided in a third embodiment of the present invention is shown. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0025] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0026] Embodiment 1

[0027] This embodiment provides a fine-tuning method based on a large model. Figure 1 A flow chart of the large model-based fine-tuning method is shown, as Figure 1 As shown, the fine-tuning method based on the large model may include steps S101 to S104, wherein:

[0028] Step S101, obtaining a first data set and a base class large model for detecting a specific object in the first data set, wherein each image in the first data set contains a specific object and a non-specific object;

[0029] The first data set is a basic image set obtained by the user to achieve the detection task of a specific object. The image content in the first data set at least includes the specific object to be detected, and may also include information unrelated to the specific object. There is no limit on the number of images in the first data set.

[0030] The base model can be a transform neural network model or other network models with similar execution principles, and there is no limitation here.

[0031] Step S102, obtaining a prompt phrase for representing a specific object category, inputting the prompt phrase into a base class large model for recognition, and obtaining a target category of the specific object;

[0032] The prompt phrases may include names used to describe specific object categories, synonyms of names, foreign language explanations of names, etc., and may also include words created by the user. For example, if a user needs to detect chairs in a group of pictures, the prompt phrases required may be "chair", "stool", and "chair".

[0033] Input a prompt for a specific object into the system, use the base class model to detect the test image based on the input prompt, and obtain the benchmark detection effect of the base class model on the specific target under each prompt. Select the prompt with the best detection effect from multiple prompts as the category name of this type of target.

[0034] Step S103, matching training fine-tuning parameters with target categories of specific objects, inserting the training fine-tuning parameters into the structural layer of the base class large model, and generating a model to be trained, wherein the training fine-tuning parameters include a low-rank matrix scaling factor, a rank of the low-rank matrix, an overfitting control factor, an inserted module, and a layer name;

[0035] The pre-trained base class large model often covers a huge amount of data, and there is usually a shortage of storage and computing resources during the application process. After the fine-tuning parameters are trained by the target category matching of a specific object, this embodiment inserts the fine-tuning parameters into the structural layer of the base class large model to participate in the training process. It is possible to insert a fine-tuning layer with the same effect as the LoRA layer without modifying the internal structure of the multi-head attention layer, saving computing resources and storage resources during the training process. Through fine-tuning technology, these models are allowed to adapt to specific application scenarios or tasks, improving their performance in specific scenarios and the detection accuracy of the base class large model for specific objects.

[0036] Before inserting the training fine-tuning parameters into the structural layer of the base class large model, initialize the environment required for the fine-tuning task, including data loading, accumulating large model structure generation, fine-tuning layer initialization and insertion, etc. After initialization, the fine-tuning layer will be inserted into the base class large model, the base class large model parameters will be frozen, and the parameters of the fine-tuning layer will be set to a trainable state. These trainable fine-tuning layers are fine-tuning adapters.

[0037] Step S104, obtaining a second data set containing only specific objects of the target category, and training the to-be-trained model using the second data set to achieve fine-tuning of the large model.

[0038] Since the target category is the data with the best prediction effect for the base class model, based on this, collecting pictures that only contain specific objects of the target category can enhance the detection effect of the model to be trained.

[0039] Specifically, the collected pictures can be 200, and the training set and the test set can be divided according to 4:1 or 5:1. The collected pictures are annotated, and the annotation data format is the coco data set format, where the annotation information includes the annotation box information and category information of the specific object, and the annotation box information includes the absolute value of the x coordinate of the upper left corner point, the absolute value of the y coordinate of the upper left corner point, the absolute value of the width of the target box, and the absolute value of the height of the target box.

[0040] Preferably, step S102 may include steps S1021 to S1024, wherein:

[0041] Step S1021, using each prompt word in the prompt word group as input data of the base class large model to perform image recognition, and obtaining all recognition objects corresponding to each prompt word in the first data set;

[0042] Step S1022, obtaining a preset standard object image set, traversing the preset object image set, and determining a standard object image corresponding to the prompt phrase;

[0043] The standard object image set may be a historical image set containing only specific objects. The standard object image set contains multiple types of standard object images. The type name may be defined according to the object name, for example, a chair, a fire extinguisher, a cup, and the like.

[0044] Step S1023, performing similarity matching between the standard object image and all the recognition objects corresponding to each prompt word, and taking the recognition objects with similarity greater than or equal to a first preset threshold as the specific objects of the prompt word;

[0045] The first preset threshold is determined by the scenario requirements and is not limited here.

[0046] Step S1024, calculating the number of specific objects for each prompt word, and taking the prompt word with the largest number of specific objects as the target category of the specific object.

[0047] The prompt phrase is used as the input data of the base class large model, and the detection effect of the same specific object of different categories based on the base class large model is achieved. Among multiple prompt words, the prompt word with the best detection effect (the largest number of specific objects detected) is selected as the category name of this type of target, which is convenient for subsequent users to accurately define the detection of specific objects, thereby achieving the recognition accuracy of the base class large model.

[0048] Use the base class large model to perform a detection benchmark test on the test image to obtain the benchmark detection effect of the base class large model on the specific target.

[0049] Preferably, step S103 may include steps S1031 to S1034, wherein:

[0050] Step S1031, obtaining a preset model fine-tuning rule table, wherein the preset model fine-tuning rule table includes a project name and a fine-tuning parameter, and the project name corresponds to the fine-tuning parameter;

[0051] The project name is the standard class name for the specific object.

[0052] Step S1032, performing text similarity matching between the target category of the specific object and the project name, if there is a project name with a text similarity greater than or equal to a second preset threshold, determining the project name as a training fine-tuning parameter for the specific object;

[0053] Step S1033, if there is no project name with a text similarity greater than or equal to the second preset threshold, semantic similarity matching is performed between the target category of the specific object and the project name, and if there is a project name with a semantic similarity greater than or equal to the third preset threshold, the project name is determined to be a training fine-tuning parameter for the specific object;

[0054] Step S1034, if there is no project name with a semantic similarity greater than or equal to the third preset threshold, obtain the manually set fine-tuning parameters as the training fine-tuning parameters of the specific object, and store the target category of the specific object and the manually set fine-tuning parameters in the preset model fine-tuning rule table.

[0055] Since the acquired prompt words do not have a unified description format, if the target category of a specific object is directly used as the search unit and the preset model fine-tuning rule table is traversed, a "no search object" situation may occur, and the fine-tuning of the base class large model cannot be achieved. This embodiment aims to determine the fine-tuning parameters that match the specific object by matching the target category of the specific object with the project name in terms of text similarity and semantic similarity. At the same time, when the fine-tuning parameters cannot be determined after the above two retrieval methods, the manually set fine-tuning parameters are used as the training fine-tuning parameters of the specific object to provide data support for the subsequent training process of the base class large model, so as to reduce the computing resources occupied by the training process.

[0056] Preferably, the training fine-tuning parameters include a plurality of fine-tuning parameter sets, and step S104 may include steps S1041 to S1044, wherein:

[0057] Step S1041, freezing the structural parameters of the base class large model, and selecting a fine-tuning parameter set from the training fine-tuning parameters as a fine-tuning layer, to generate multiple first models to be trained;

[0058] Step S1042, inputting the second data set into the first model to be trained for training, performing forward propagation calculation through each layer of the model, and outputting the predicted category of the specific object of the second data set;

[0059] Step S1043, respectively calculating the loss function value between the predicted category of the first to-be-trained model corresponding to each fine-tuning parameter set and the true label of the second data set;

[0060] Step S1044, determine the target fine-tuning parameter set through the loss function value, and construct a fine-tuning large model with the fine-tuning layer composed of the target fine-tuning parameter set and the base class large model.

[0061] Specifically, the target fine-tuning parameter set is determined by the loss function value, including step A1 to step A2, wherein:

[0062] Step A1, comparing the values ​​of different loss functions, and determining the fine-tuning parameter set corresponding to the minimum loss function value as the target fine-tuning parameter set;

[0063] Step A2: Save the target fine-tuning parameter set in the form of a model weight file in the database.

[0064] After full fine-tuning of the base class large model, the output model weight size may reach hundreds of MB or even GB, but the model weight file is only a few MB, which can greatly save storage resources.

[0065] In addition, the training of the base class large model with the fine-tuning layer inserted is started. During this process, the user can choose appropriate training parameters (such as learning rate, number of training iterations, number of input batch training images, etc.) to fine-tune the training of the new large model.

[0066] Preferably, after obtaining a second data set containing only specific objects of the target category and training the to-be-trained model by using the second data set to achieve fine-tuning of the large model, the method further includes steps B1 to B5, wherein:

[0067] Step B1, in response to a user's detection operation on multiple specific objects, obtaining prompt words for multiple specific objects;

[0068] Step B2, obtaining a preset reasoning configuration table, wherein the preset reasoning configuration table includes a prompt word, a model weight file name, a model weight file path, a large model output result filtering threshold, and a label sequence number;

[0069] There is a unique mapping relationship between the prompt word, the model weight file name, the model weight file path, the large model output result filtering threshold and the label sequence number.

[0070] Step B3, determining the model weight file name, model weight file path, large model output result filtering threshold and label sequence number of each specific object in the preset reasoning configuration table through the prompt words of multiple specific objects;

[0071] Step B4, determining the target model weight file by the model weight file name and / or the model weight file path;

[0072] Step B5, load the target model weight file and the large model output result filtering threshold into the base class large model according to the label sequence number to realize parallel reasoning of multiple specific objects.

[0073] In actual business scenarios, users may need to implement task reasoning for multiple specific objects. In the traditional fine-tuning reasoning method, a fine-tuning adapter + a large base class model is only valid for a specific task. For multiple tasks, multiple fine-tuning adapters + multiple large base class models are required to take effect in series. This method will take up more computing resources and is not conducive to the effective execution of the training process. In this embodiment, multiple fine-tuning models are loaded for parallel reasoning by inserting multiple fine-tuning adapters into a large base class model at the same time, and with very little increase in video memory resources, one reasoning can be effective for multiple specific tasks at the same time.

[0074] The following is an example of actual user operation:

[0075] After model deployment, the system builds a large model for inference based on the number of detection tasks entered by the user, and infers the data sent in the task scenario. Users can set the following parameters to build the inference large model:

[0076] a.num_adapters: the number of fine-tuning adapters loaded

[0077] b.name: the name of each fine-tuning adapter

[0078] c.model_path: each fine-tuning adapter model weight path

[0079] d.prompt: The prompt word for each specific task goal

[0080] e.threshold: The filtering threshold for each large model output result

[0081] f.label_num: label number used for each target

[0082] After the model is deployed and starts processing user data, the results of problematic model inference reported by users can be collected and added to the training dataset for fine-tuning adapter training iterations for specific tasks.

[0083] This embodiment determines the target category of the feature object through the prompt phrase of the feature object, further matches the corresponding training fine-tuning parameters according to the target category, and only inserts the fine-tuning parameters into the base class large model for data adjustment during the training process, thereby reducing the computing resource occupancy and data volume of the base class large model in the training and reasoning process of specific object category detection, thereby saving computing resources.

[0084] Embodiment 2

[0085] Embodiment 2 of the present invention further provides a fine-tuning device based on a large model, which corresponds to the fine-tuning method based on a large model provided in the above embodiment 1. The corresponding technical features and technical effects are not described in detail in this embodiment, and the relevant parts can refer to the above embodiment 1. Specifically, Figure 2 The structural block diagram of the fine-tuning device based on the large model is shown. Figure 2 As shown, the fine-tuning device 200 based on the large model includes an acquisition module 201, a recognition module 202, a fine-tuning module 203 and a training module 204, wherein:

[0086] An acquisition module 201 is used to acquire a first data set and a base class macro model for detecting a specific object in the first data set, wherein each image in the first data set contains a specific object and a non-specific object;

[0087] The recognition module 202 is connected to the acquisition module 201 and is used to acquire a prompt phrase for representing a specific object category, input the prompt phrase into the base class model for recognition, and obtain a target category of the specific object;

[0088] The fine-tuning module 203 is connected to the recognition module 202 and is used to match the training fine-tuning parameters with the target category of the specific object, insert the training fine-tuning parameters into the structural layer of the base class large model, and generate a model to be trained, wherein the training fine-tuning parameters include a low-rank matrix scaling factor, a rank of the low-rank matrix, an overfitting control factor, an inserted module and a layer name;

[0089] The training module 204 is connected to the fine-tuning module 203 and is used to obtain a second data set containing only specific objects of the target category, and train the to-be-trained model through the second data set to achieve fine-tuning of the large model.

[0090] Optionally, the recognition module is specifically used to: use each prompt word in the prompt word group as input data of the base class large model for image recognition, and obtain all recognition objects corresponding to each prompt word in the first data set; obtain a preset standard object image set, traverse the preset object image set, and determine the standard object image corresponding to the prompt word; perform similarity matching on the standard object image and all recognition objects corresponding to each prompt word, and use the recognition objects with a similarity greater than or equal to a first preset threshold as specific objects of the prompt word; calculate the number of specific objects for each prompt word, and use the prompt word with the largest number of specific objects as the target category of the specific object.

[0091] Optionally, the fine-tuning module is specifically used to: obtain a preset model fine-tuning rule table, wherein the preset model fine-tuning rule table includes a project name and fine-tuning parameters, and the project name corresponds to the fine-tuning parameters; perform text similarity matching on the target category of the specific object and the project name, and if there is a project name whose text similarity is greater than or equal to a second preset threshold, determine that the project name is a training fine-tuning parameter for the specific object; if there is no project name whose text similarity is greater than or equal to the second preset threshold, perform semantic similarity matching on the target category of the specific object and the project name, and if there is a project name whose semantic similarity is greater than or equal to a third preset threshold, determine that the project name is a training fine-tuning parameter for the specific object; if there is no project name whose semantic similarity is greater than or equal to the third preset threshold, obtain the manually set fine-tuning parameters as the training fine-tuning parameters for the specific object, and store the target category of the specific object and the manually set fine-tuning parameters in the preset model fine-tuning rule table.

[0092] Optionally, the training module is also used to: freeze the structural parameters of the base class large model, and select a fine-tuning parameter set from the training fine-tuning parameters as the fine-tuning layer to generate multiple first models to be trained; input the second data sets into the first models to be trained for training, perform forward propagation calculations through each layer of the model, and output the predicted category of the specific object of the second data set; calculate the loss function value between the predicted category of the first model to be trained corresponding to each fine-tuning parameter set and the true label of the second data set; determine the target fine-tuning parameter set through the loss function value, and construct the fine-tuning large model with the fine-tuning layer composed of the target fine-tuning parameter set and the base class large model.

[0093] Optionally, the training module is also used to: compare the sizes of different loss function values, determine the fine-tuning parameter set corresponding to the minimum loss function value as the target fine-tuning parameter set; and save the target fine-tuning parameter set in the form of a model weight file in the database.

[0094] Optionally, the device also includes a parallel reasoning module, which is used to: obtain prompt words for multiple specific objects in response to the user's detection operation on multiple specific objects; obtain a preset reasoning configuration table, wherein the preset reasoning configuration table includes prompt words, model weight file name, model weight file path, large model output result filtering threshold and label serial number; determine the model weight file name, model weight file path, large model output result filtering threshold and label serial number of each specific object in the preset reasoning configuration table through the prompt words of multiple specific objects; determine the target model weight file through the model weight file name and / or model weight file path; load the target model weight file and the large model output result filtering threshold into the base class large model according to the label serial number to realize parallel reasoning for multiple specific objects.

[0095] Embodiment 3

[0096] Figure 3 The block diagram of a computer device suitable for implementing a fine-tuning method based on a large model provided in the third embodiment of the present invention is shown. In this embodiment, the computer device 300 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a rack server, a blade server, a tower server or a cabinet server (including an independent server or a server cluster composed of multiple servers) that executes a program. Figure 3 As shown, the computer device 300 of this embodiment includes at least but is not limited to: a memory 301, a processor 302, and a network interface 303 that can be interconnected through a system bus. It should be noted that Figure 3 Only computer device 300 having components 301 - 303 is shown, but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0097] In this embodiment, the memory 303 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 301 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 301 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 301 can also include both the internal storage unit of the computer device 300 and its external storage device. In this embodiment, the memory 301 is generally used to store an operating system and various application software installed on the computer device 300, such as program codes of a fine-tuning method based on a large model.

[0098] In some embodiments, the processor 302 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 302 is generally used to control the overall operation of the computer device 300. For example, the processor 302 performs control and processing related to data interaction or communication with the computer device 300. In this embodiment, the processor 302 is used to run the program code of the steps of the fine-tuning method based on the large model stored in the memory 301.

[0099] In this embodiment, the large model-based fine-tuning method stored in the memory 301 can also be divided into one or more program modules and executed by one or more processors (processor 302 in this embodiment) to complete the present invention.

[0100] The network interface 303 may include a wireless network interface or a wired network interface, and the network interface 303 is generally used to establish a communication link between the computer device 300 and other computer devices. For example, the network interface 303 is used to connect the computer device 300 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 300 and the external terminal. The network can be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0101] Embodiment 4

[0102] This embodiment also provides a computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., on which a computer program is stored, and when the computer program is executed by a processor, the steps of the fine-tuning method based on the large model are implemented.

[0103] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0104] It should be noted that the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0105] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0106] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A fine-tuning method based on a large model, characterized in that: The method comprises: Acquire a first data set and a base class macro model for detecting a specific object in the first data set, wherein each picture in the first data set contains a specific object and non-specific objects; Obtaining a prompt phrase for representing a specific object category, inputting the prompt phrase into the base class large model for recognition, and obtaining a target category of the specific object; Matching training fine-tuning parameters with the target category of the specific object, inserting the training fine-tuning parameters into the structural layer of the base class large model, and generating a model to be trained, wherein the training fine-tuning parameters include a low-rank matrix scaling factor, a rank of the low-rank matrix, an overfitting control factor, an inserted module and a layer name; A second data set containing only specific objects of the target category is obtained, and the model to be trained is trained using the second data set to achieve fine-tuning of the large model.

2. The method according to claim 1, characterized in that: The step of inputting the prompt phrase into the base class large model for identification to obtain the target category of the specific object includes: Using each prompt word in the prompt word group as input data of the base class large model to perform image recognition, and obtaining all recognition objects corresponding to each prompt word in the first data set; Obtaining a preset standard object image set, traversing the preset object image set, and determining a standard object image corresponding to the prompt phrase; Performing similarity matching between the standard object image and all the recognition objects corresponding to each prompt word, and taking the recognition objects with a similarity greater than or equal to a first preset threshold as the specific objects of the prompt word; The number of specific objects for each cue word is calculated, and the cue word with the largest number of specific objects is taken as the target category of the specific object.

3. The method according to claim 1 or 2, characterized in that: The training and fine-tuning parameters by target category matching of the specific object include: Obtaining a preset model fine-tuning rule table, wherein the preset model fine-tuning rule table includes a project name and a fine-tuning parameter, and the project name corresponds to the fine-tuning parameter; Performing text similarity matching between the target category of the specific object and the project name, and if there is a project name whose text similarity is greater than or equal to a second preset threshold, determining the project name as a training fine-tuning parameter of the specific object; If there is no project name whose text similarity is greater than or equal to the second preset threshold, semantic similarity matching is performed between the target category of the specific object and the project name; if there is a project name whose semantic similarity is greater than or equal to the third preset threshold, the project name is determined to be a training fine-tuning parameter of the specific object; If there is no project name with a semantic similarity greater than or equal to the third preset threshold, obtain the manually set fine-tuning parameters as the training fine-tuning parameters of the specific object, and store the target category of the specific object and the manually set fine-tuning parameters in the preset model fine-tuning rule table.

4. The method according to claim 1, characterized in that: The training fine-tuning parameters include a plurality of fine-tuning parameter sets, and the training of the model to be trained by using the second data set to achieve fine-tuning of the large model includes: Freeze the structural parameters of the base class large model, and select a fine-tuning parameter set from the training fine-tuning parameters as a fine-tuning layer to generate a plurality of first models to be trained; Inputting the second data set into the first model to be trained for training, performing forward propagation calculation through each layer of the model, and outputting the predicted category of the specific object of the second data set; Calculate the loss function value between the predicted category of the first to-be-trained model corresponding to each fine-tuning parameter set and the true label of the second data set respectively; The target fine-tuning parameter set is determined by the loss function value, and a fine-tuning large model is constructed using the fine-tuning layer and the base class large model composed of the target fine-tuning parameter set.

5. The method according to claim 4, characterized in that The determining of the target fine-tuning parameter set by the loss function value includes: Compare the values ​​of different loss functions, and determine the fine-tuning parameter set corresponding to the minimum loss function value as the target fine-tuning parameter set; The target fine-tuning parameter set is saved in the database in the form of a model weight file.

6. The method according to claim 1, characterized in that After obtaining a second data set containing only specific objects of the target category and training the model to be trained by using the second data set to achieve fine-tuning of the large model, the method further includes: In response to a user's detection operation on a plurality of specific objects, obtaining prompt words for the plurality of specific objects; Obtain a preset reasoning configuration table, wherein the preset reasoning configuration table includes a prompt word, a model weight file name, a model weight file path, a large model output result filtering threshold, and a label sequence number; Determine the model weight file name, model weight file path, large model output result filtering threshold and label sequence number of each specific object in the preset reasoning configuration table through the prompt words of multiple specific objects; Determine the target model weight file by the model weight file name and / or the model weight file path; The target model weight file and the large model output result filtering threshold are loaded into the base class large model according to the label serial number to realize parallel reasoning of multiple specific objects.

7. A fine-tuning device based on a large model, characterized in that: The device comprises: An acquisition module, used for acquiring a first data set and a base class macro model for detecting a specific object in the first data set, wherein each image in the first data set contains a specific object and non-specific objects; A recognition module, used to obtain a prompt phrase for characterizing a specific object category, input the prompt phrase into the base class large model for recognition, and obtain a target category of the specific object; A fine-tuning module, used for matching training fine-tuning parameters with the target category of the specific object, inserting the training fine-tuning parameters into the structural layer of the base class large model, and generating a model to be trained, wherein the training fine-tuning parameters include a low-rank matrix scaling factor, a rank of the low-rank matrix, an overfitting control factor, an inserted module and a layer name; The training module is used to obtain a second data set containing only specific objects of the target category, and train the model to be trained through the second data set to achieve fine-tuning of the large model.

8. The device according to claim 7, characterized in that The device also includes a parallel reasoning module, which is used to: In response to a user's detection operation on a plurality of specific objects, obtaining prompt words for the plurality of specific objects; Obtain a preset reasoning configuration table, wherein the preset reasoning configuration table includes a prompt word, a model weight file name, a model weight file path, a large model output result filtering threshold, and a label sequence number; Determine the model weight file name, model weight file path, large model output result filtering threshold and label sequence number of each specific object in the preset reasoning configuration table through the prompt words of multiple specific objects; Determine the target model weight file by the model weight file name and / or the model weight file path; The target model weight file and the large model output result filtering threshold are loaded into the base class large model according to the label serial number to realize parallel reasoning of multiple specific objects.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Sensitive word auditing method based on large language model, storage medium and electronic equipment

    CN116720515A

  • Large model fine tuning method, device and equipment based on Lora layer number combination and medium

    CN117910531A

  • Large-model low-rank adaptive fine tuning method and device based on tensor parallelism, and medium

    CN117993467A

  • Training method of fine tuning model and related product

    CN118227018A

  • Source code vulnerability detection method and system based on large model

    CN118332557A