Method, device and medium for fine-tuning of pre-trained model

Abstract

The application relates to the fields of artificial intelligence and medical health, and discloses a fine-tuning method of a pre-training model, which comprises the following steps: acquiring a pre-training model, setting a parameter fine-tuning module in each layer of the pre-training model, and obtaining a to-be-fine-tuned model; acquiring a data set corresponding to a target task, adjusting the architecture and parameters of each parameter fine-tuning module in the to-be-fine-tuned model according to the data set by using a differentiable architecture search method, and obtaining a target parameter fine-tuning module corresponding to the parameter fine-tuning module; and combining the pre-training model and the target parameter fine-tuning module to obtain an adjusted model corresponding to the target task. The method solves the problem that the existing parameter efficient fine-tuning method has poor fine-tuning effect due to simple architecture and cannot perform medical field tasks.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence technology and healthcare, and in particular to a method, apparatus, device, and medium for fine-tuning a pre-trained model. Background Technology

[0002] A pre-trained model is a model that has been trained on a large-scale dataset in advance. Using this model on the target task can reduce the task execution cost and improve the task efficiency.

[0003] Medical data is typically highly complex and diverse. For example, medical data may contain noise, missing values, or random variations, resulting in low model generality for each task and requiring tailored model designs for each task. Therefore, when using pre-trained models in the medical field, the initial model needs to be fine-tuned to suit the characteristics of the target task. However, with the development of pre-trained language models, the number of language model parameters has become increasingly large, typically exceeding 300 million or even reaching 1 billion. Consequently, fine-tuning and deploying models individually for each task is extremely costly.

[0004] To address the aforementioned issues, efficient parameter fine-tuning methods are crucial. These methods require only fine-tuning a few million parameters for each different task (dataset), enabling the adaptation of large pre-trained models to the task and achieving performance essentially the same as or very close to that of fine-tuning the entire model using all its parameters.

[0005] Adapter is a typical method for efficient parameter fine-tuning. However, the existing Adapter method has a relatively simple architecture and cannot effectively mine the features provided by the pre-trained model. Therefore, its fine-tuning effect is worse than that of full parameter fine-tuning of the pre-trained model. It is not suitable for medical tasks with high complexity and high requirements for reliability and accuracy. Summary of the Invention

[0006] In view of this, this application provides a method, apparatus, medium and device for fine-tuning a pre-trained model, which solves the problem that the existing efficient parameter fine-tuning methods have poor fine-tuning effects due to their simple architecture, making them unsuitable for tasks in the medical field.

[0007] A first aspect of this application provides a method for fine-tuning a pre-trained model, the method comprising:

[0008] Obtain a pre-trained model and set a parameter fine-tuning module in each layer of the pre-trained model to obtain the model to be fine-tuned;

[0009] Obtain the dataset corresponding to the target task, and in the model to be fine-tuned, use the differentiable architecture search method to adjust the architecture and parameters of each parameter fine-tuning module according to the dataset to obtain the target parameter fine-tuning module corresponding to the parameter fine-tuning module;

[0010] By combining the pre-trained model and the target parameter fine-tuning module, an adjusted model corresponding to the target task is obtained.

[0011] A second aspect of this application provides a fine-tuning apparatus for a pre-trained model, the apparatus comprising:

[0012] An initialization module is used to obtain a pre-trained model and set a parameter fine-tuning module in each layer of the pre-trained model to obtain the model to be fine-tuned.

[0013] The fine-tuning module obtains the dataset corresponding to the target task, and in the model to be fine-tuned, uses a differentiable architecture search method to adjust the architecture and parameters of each parameter fine-tuning module according to the dataset to obtain the target parameter fine-tuning module corresponding to the parameter fine-tuning module;

[0014] The combination module is used to combine the pre-trained model and the target parameter fine-tuning module to obtain the adjusted model corresponding to the target task.

[0015] A third aspect of this application provides an apparatus including a storage medium, a processor, and instructions or code stored on the storage medium and executable on the processor, wherein the processor executes the instructions or code to implement the fine-tuning method of the pre-trained model described above.

[0016] In a fourth aspect, this application provides a medium having instructions or code stored thereon, which, when executed by a processor, implement the fine-tuning method of the pre-trained model described above.

[0017] The aforementioned method, apparatus, device, and medium for fine-tuning pre-trained models adds a parameter fine-tuning module to each layer of the pre-trained model. During model training, only the parameters in the parameter fine-tuning module are updated, without adjusting the parameters of the main pre-trained model. Since the parameter size of the fine-tuning module is much smaller than that of the pre-trained model, the computational cost of adjusting its parameters is relatively small. Compared to methods that adjust all parameters of the pre-trained model, this method of adjusting only the parameter fine-tuning module requires only fine-tuning a relatively small number of parameters to adapt a large pre-trained model to a specific task. This reduces the demand for computational resources and saves training time. Furthermore, by using a differentiable architecture search method to adjust the architecture of the parameter fine-tuning module for each layer, different adapter architectures are selected for layers of different depths in the pre-trained model, resulting in a target parameter fine-tuning module that is more suitable for the corresponding layer. This approach optimizes the architecture, solving the problem of low model quality caused by the simple architecture of existing efficient parameter fine-tuning modules. It is suitable for tasks with high complexity and high requirements for reliability and accuracy, such as those in the medical field.

[0018] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 The diagram shows a flowchart of a fine-tuning method for a pre-trained model provided in an embodiment of this application.

[0021] Figure 2 A schematic diagram of the model structure to be fine-tuned is shown in another fine-tuning method for a pre-trained model provided in the embodiments of this application.

[0022] Figure 3 A flowchart illustrating another fine-tuning method for a pre-trained model provided in an embodiment of this application is shown.

[0023] Figure 4 The present application provides a schematic diagram of the parameter fine-tuning module structure for another pre-trained model fine-tuning method according to an embodiment of the present application.

[0024] Figure 5 A schematic diagram of a differentiable architecture search for another fine-tuning method for a pre-trained model provided in this application embodiment is shown;

[0025] Figure 6 The present application provides a structural block diagram of a fine-tuning device for a pre-trained model according to an embodiment;

[0026] Figure 7 The present application provides a structural block diagram of an electronic device according to an embodiment. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The fine-tuning method for pre-trained models provided in this application can be applied to electronic devices with instruction or program execution capabilities. These electronic devices can be, but are not limited to, various servers, workstations, personal computers, laptops, etc. The difference in execution between different computing devices is merely a difference in the execution entity; those skilled in the art can foresee that running the solution on different computing devices will produce the same technical effect. The invention will now be described in detail through specific embodiments.

[0029] Please see Figure 1 As shown, Figure 1 A flowchart illustrating the fine-tuning method for a pre-trained model provided in this embodiment of the invention includes the following steps:

[0030] S101: Obtain the pre-trained model and set a parameter fine-tuning module in each layer of the pre-trained model to obtain the model to be fine-tuned.

[0031] The method provided by this invention can be used for parameter fine-tuning of pre-trained models, addressing the problem of low model quality caused by the simple architecture of existing efficient parameter fine-tuning modules. In practical applications, a pre-trained model is first obtained, then the pre-trained model and the parameter fine-tuning module are combined to obtain the model to be fine-tuned. The model to be fine-tuned is then adjusted to obtain the adjusted model, which can then be used to perform corresponding tasks.

[0032] Figure 2A schematic diagram of the structure of the model to be fine-tuned according to an embodiment of this application is shown. As shown in the figure, the PLM module (pretrained language model) is the backbone of the pretrained model. For example, in pathological analysis, the backbone of the pretrained pathological analysis model has several transformer layers. A parameter fine-tuning module (such as an adapter module) is added to each transformer layer. The adapter module is added after the linear connection layer of the transformer. It can be understood that the adapter module is a commonly used, efficient parameter fine-tuning module. After receiving the intermediate vector representation H of a sample x from the pretrained model backbone, the adapter module first performs a dimensionality reduction operation W. down Get H (1) Then, H is obtained through an activation function such as the ReLU function. (2) Then through dimensional upgrading operation W up Get H (3) Therefore, after adding the Adapter module, the pathological analysis model's representation of the sample changed from H to H+H. (3) By adding the Adapter module, especially its dimensionality reduction and expansion operations, it is possible to achieve good performance in a downstream task (such as tumor classification, fibrosis assessment, and cancer staging) by adding and fine-tuning less than 1% of the PLM parameters.

[0033] S102: Obtain the dataset corresponding to the target task, and in the model to be fine-tuned, use the differentiable architecture search method to adjust the architecture and parameters of each parameter fine-tuning module according to the dataset to obtain the target parameter fine-tuning module corresponding to the parameter fine-tuning module.

[0034] In this step, machine learning methods are used to automatically learn the parameter fine-tuning modules for each layer, thereby updating the parameter fine-tuning modules and obtaining the target parameter fine-tuning modules. It is understandable that, since each layer of the pre-trained model extracts features at different levels from the input data such as sentences or images, each layer's parameter fine-tuning module may be more suitable for different architectures. Therefore, in addition to parameter updates, the update of the parameter fine-tuning modules also includes architecture updates.

[0035] Differentiable Architecture Search (DARTS) can be used to update the parameter fine-tuning module. DARTS is a method for automatically searching neural network architectures. Its core idea is to transform the network architecture search process into a differentiable optimization problem. It introduces a set of learnable hyperparameters to represent the continuous space between different network structures. By using optimization methods such as gradient descent, DARTS can iteratively update these hyperparameters on the training set, thereby searching for the optimal network architecture. Specifically, the search space in DARTS includes a complete set of candidate operations, and different operation structures are combined in a certain proportion using mechanisms such as weight sharing and weighted summation. Thus, in each training iteration, DARTS automatically learns the optimal operation structure for the current iteration by optimizing the hyperparameter weights, thus obtaining the architecture of the target parameter fine-tuning module.

[0036] In practical applications, medical data for tasks such as pathological analysis, medical image analysis, and disease prediction differ, thus requiring different network architectures. Using differentiable architecture search methods to update parameter fine-tuning modules can quickly find network architectures suitable for the characteristics of different medical data. It's understandable that each task has its unique goals and requirements. Even within pathological analysis tasks, the requirements for tumor classification, fibrosis assessment, and cancer staging differ. Therefore, training on datasets corresponding to the target task yields parameter fine-tuning modules that are more suitable for solving specific tasks.

[0037] S103: Combine the pre-trained model and the target parameter fine-tuning module to obtain the adjusted model corresponding to the target task.

[0038] In this step, the pre-trained model and the updated target parameter fine-tuning module are combined to obtain the adjusted model. Specifically, the parameter fine-tuning module of each layer in the model to be adjusted is replaced with the corresponding target parameter fine-tuning module.

[0039] This embodiment adds a parameter fine-tuning module to each layer of the pre-trained model. During model training, only the parameters in the parameter fine-tuning module are updated, without adjusting the parameters of the main pre-trained model. Since the parameter size of the parameter fine-tuning module is much smaller than that of the pre-trained model, the computational cost of adjusting its parameters is also relatively small. Compared to methods that adjust all parameters of the pre-trained model, this method of adjusting only the parameter fine-tuning module requires only fine-tuning a relatively small number of parameters to adapt a large pre-trained model to a specific task. This reduces the demand for computational resources and saves training time. Furthermore, a differentiable architecture search method is used to adjust the architecture of the parameter fine-tuning module for each layer, thereby selecting different adapter architectures for layers of different depths in the pre-trained model. The resulting target parameter fine-tuning module is more suitable for the corresponding layer. This approach optimizes the architecture, solving the problem of low model quality caused by the simple architecture of existing efficient parameter fine-tuning modules. It is suitable for tasks with high complexity and high requirements for reliability and accuracy, such as those in the medical field.

[0040] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, other fine-tuning methods for pre-trained models are provided, such as... Figure 3 As shown, another method for fine-tuning a pre-trained model includes the following steps:

[0041] S201: Obtain training data and use the training data to train a preset neural network model to obtain a pre-trained model.

[0042] In this step, a pre-trained model is obtained by training the initial neural network model using a large amount of data. The training data is used to train, validate, or test the model, and can be manually labeled text data, image data, or other forms of data. The training data is used to provide the input and output samples required for the model's training.

[0043] For example, in the medical field, training data can be images of tissue samples or tissue slices, clinical data, blood cell images, or blood test data, etc.

[0044] S202: Set a parameter fine-tuning module in each layer of the pre-trained model to obtain the model to be fine-tuned.

[0045] S203: Obtain the dataset corresponding to the target task.

[0046] S204: In each parameter fine-tuning module, a supernet is constructed based on the dataset, and a target subnet is selected in the supernet, which is then used as the architecture of the parameter fine-tuning module.

[0047] S205: Adjust the parameters of the target sub-network using the dataset to obtain the target parameter fine-tuning module, wherein the parameters of the target sub-network include at least the activation function parameters and the encoder parameters.

[0048] In steps S204-S205, a differentiable architecture search method is used to update the parameter fine-tuning module. Specifically, a supernet is first constructed based on the dataset. The supernet can contain multiple different subnetworks, each with different architectural features, such as different numbers of layers, different convolutional structures, or different connection methods.

[0049] After constructing the supernet, the supernet can be searched through training to select the nearest subnetwork structure, which is the target subnetwork of the layer where the parameter fine-tuning module is located, and this subnetwork is used as the architecture of the parameter fine-tuning module.

[0050] After obtaining the target subnetwork, training can continue to further optimize its parameters, resulting in the final target parameter fine-tuning module. During parameter optimization, the activation function and encoder can be optimized. For example, the activation function can be one of RELU (Rectified Linear Units), GELU (Gaussian Error Linear Unit), SWISH (Self-gated activation function), or ELU (Exponential Linear Unit), while the kernel size in the encoder can be selected as 1, 3, 5, etc.

[0051] Figure 4 A schematic diagram of the parameter fine-tuning module according to an embodiment of this application is shown in the figure. Figure 4 The left half of the diagram shows the architecture of the Adapter module. Figure 4 The right half of the diagram contains the encoder cell and the activation cell. The architecture of the Adapter module includes sequentially connected dimensionality reduction units, activation function units, encoder units, activation function units, and dimensionality increase units. In the process of improving the Adapter unit, in addition to the dimensionality reduction and dimensionality increase units, the activation function must also be determined. For example, a more suitable activation function must be selected from the three activation functions Act1, Act2, and Act3. In addition, it must be determined whether to add encoder units that perform more in-depth feature extraction, such as Op1, Op2, and Op3 in the diagram.

[0052] This embodiment constructs a supernet containing multiple subnets, which can simultaneously search multiple different network architectures and quickly find the optimal network architecture, i.e., the target subnet, that is, suitable for a specific layer. Based on this, the parameters of the target subnet are further adjusted to obtain a parameter fine-tuning module that is more suitable for each layer, thereby improving the performance and generalization ability of the parameter fine-tuning module.

[0053] In step S204, a supernet is constructed based on the dataset, and a target subnet is selected within the supernet, including:

[0054] S2041: Construct a supernet, wherein the supernet comprises multiple layers of network modules, and each layer of network modules comprises a set of basic operations.

[0055] In this context, the edges of the hypernet are multiple basic operations in the set of basic operations, and the nodes of the hypernet are the hidden state features of the data in the dataset after undergoing basic operations.

[0056] The basic operation set includes at least one of the following operations: convolution operation, self-attention connection operation, fully connected operation, and residual connection operation.

[0057] S2042: Train the supernet using the dataset, obtain the updated supernet parameters, and search for the target subnetwork in the supernet based on the updated supernet parameters.

[0058] In steps S2041-S2041, a supernet is first constructed, and then an architecture search method is used to search for the optimal network model architecture, i.e., the target sub-network, within the supernet. Specifically, the supernet has multiple layers of network blocks, such as dimensionality-upgrading units, dimensionality-reducing units, and activation function units. Each network block is composed of multiple basic operations connected together. These basic operations refer to the fundamental operations of the neural network, which can be viewed as functions, such as convolution operations (e.g., 3×3 convolution, 1×1 convolution, 5×5 convolution), self-attention connection operations, fully connected operations, residual connection operations, etc. In the supernet, each node (e.g., ... Figure 5 In Figure (a), nodes 0, 1, 2, and 3 represent the hidden states after the basic network operations, and the connections (edges) between each node represent the basic operation o(x). In the supernet, the predecessor node i performs feature extraction through all operations in the operation set, and outputs the weighted results to node j connected to node i. Node j receives the feature extraction results of all its connected predecessor nodes, and adds all the feature extraction results to obtain the feature corresponding to node j.

[0059] Since each edge in a hypernet corresponds to multiple basic operations, the specific network structure of each edge is not fixed. Therefore, the hypernet is trained using a dataset to determine which basic operation is used for each edge, thus obtaining the target subnet. In the specific operation process, the hypernet parameters are first updated during training, for example, through backpropagation and optimization algorithms, to fine-tune the parameters and improve performance. Then, the target subnet is searched. Based on the updated hypernet parameters, the best basic operation is selected from all candidate basic operations to obtain the target subnet. For example, the subnet search can be performed by selecting the basic operation with the highest gradient value.

[0060] In step S2042, the supernet is trained using the dataset to obtain updated supernet parameters, and the target subnetwork is searched within the supernet based on the updated supernet parameters, including the following steps:

[0061] S20421: Assign the same weight to each basic operation corresponding to the edge, and calculate the output of the node connected to the edge based on the weight.

[0062] S20422: Input the dataset into the model to be fine-tuned, adjust the hypernet parameters according to the output of the nodes, and obtain the updated hypernet parameters, which include the updated weights.

[0063] S20423: Determine the target basic operation among multiple basic operations based on the updated weights.

[0064] S20424: In a supernet, remove all basic operations other than the target basic operations to obtain the target subnet.

[0065] In steps S20421-S20424, as follows Figure 5 As shown in Figure (b), suppose the connection between nodes i and j, i.e., the edge (i,j), has one of the basic operations that can be chosen from set O, where each line represents a basic operation. To determine which basic operation to choose as the target basic operation for each edge, we can perform a certain operation o on edge (i,j). (i,j) Assign a learnable weight α (i,j) Then, these basic operations are placed on this edge. The output of this edge, that is, the output of node j, is the weighted average of the results of these operations, as shown in the following formula:

[0066]

[0067] in, For some operation o on edge (i,j) (i,j) The corresponding weights.

[0068] In this context, the initial values ​​of the learnable weights are all the same. During training, the weight α... (i,j) It is also optimized and updated along with the supernet parameters. In this way, different basic operations receive different weight values. Therefore, the basic operation with the highest weight on each edge can be extracted and used as the target basic operation corresponding to that edge, such as... Figure 5 The highlighted lines shown in Figure (c) represent the selected target basic operations. Finally, in the supernet, all other basic operations besides the target basic operations are removed, leaving only the target basic operations, thus obtaining the target subnet. The final target subnet is shown below. Figure 5 As shown in Figure (d), the obtained target subnetwork can then be trained on the dataset to further optimize its parameters. This process is highly efficient, yielding a new network architecture suitable for the corresponding layer of the pre-trained model in a very short time.

[0069] This embodiment uses an architecture search method to search for and optimize the best sub-network architecture in the supernet, reducing the workload of manually designing and adjusting the network architecture. The resulting target sub-network is more suitable for the corresponding layer of the pre-trained model and is more targeted, thus achieving better performance on a given target task and dataset.

[0070] S206: Combine the pre-trained model and the target parameter fine-tuning module to obtain the adjusted model corresponding to the target task.

[0071] S207: Read the data to be processed and input the data to be processed into the adjusted model.

[0072] S208: In the adjusted model, the target task is executed on the data to be processed to obtain the processing result corresponding to the target task.

[0073] In steps S207-S208, after obtaining the adjusted model, the target task can be executed using the adjusted model. Specifically, the data to be processed for the target task is read and input into the adjusted model. The data is then processed within the model to obtain the corresponding processing results.

[0074] For example, in pathological analysis tasks, images of tumor tissue slices are input into an adjusted model as data to be processed. The model distinguishes different types of tumors, such as breast cancer, lung cancer, or colon cancer, based on the morphological and structural features of cells in the slices, and finally outputs the corresponding tumor type.

[0075] Since the adjusted model is trained based on the target task and the corresponding dataset, it is more targeted and performs better in executing the target task, resulting in better processing results compared to the unadjusted preprocessed model.

[0076] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0077] As can be seen, this solution achieves automatic learning and enhancement of the Adapter module through machine learning methods. It can select different Adapter architectures for transformer layers of varying depths, effectively improving the fine-tuning of the Adapter module and making each Adapter module more compatible with its corresponding transformer layer, thus enhancing the performance of the adjusted model when performing the target task. Building on this, a differentiable architecture search method is used to perform specific Adapter module optimization operations. This method can simultaneously search for multiple different network architectures, finding the optimal solution in a larger search space. Compared to other methods, it offers higher efficiency and flexibility, quickly finding network architectures suitable for the characteristics of medical data. Furthermore, after determining the specific architecture, the parameters of the Adapter module are further optimized, further improving the performance of the adjusted model.

[0078] In one embodiment, a fine-tuning device for a pre-trained model is provided, which corresponds one-to-one with the fine-tuning method for the pre-trained model in the above embodiments. For example... Figure 6 As shown, the fine-tuning device for this pre-trained model includes an initialization module, a fine-tuning module, and a combination module. Detailed descriptions of each functional module are as follows:

[0079] The initialization module is used to obtain the pre-trained model and set a parameter fine-tuning module for each layer of the pre-trained model to obtain the model to be fine-tuned.

[0080] The fine-tuning module obtains the dataset corresponding to the target task and uses a differentiable architecture search method in the model to be fine-tuned to adjust the architecture and parameters of each parameter fine-tuning module according to the dataset, thereby obtaining the target parameter fine-tuning module corresponding to the parameter fine-tuning module.

[0081] The combination module is used to combine the pre-trained model and the target parameter fine-tuning module to obtain the adjusted model corresponding to the target task.

[0082] In one embodiment, the fine-tuning module is used for:

[0083] In each parameter fine-tuning module, a supernet is constructed based on the dataset, and a target subnet is selected in the supernet. The target subnet is used as the architecture of the parameter fine-tuning module.

[0084] The parameters of the target sub-network are adjusted using the dataset to obtain the target parameter fine-tuning module. The parameters of the target sub-network include at least the activation function parameters and the encoder parameters.

[0085] In one embodiment, the fine-tuning module is used for:

[0086] Construct a supernet, which consists of multiple layers of network modules, each of which includes a set of basic operations.

[0087] The supernet is trained using the dataset to obtain updated supernet parameters, and the target subnetwork is searched within the supernet based on the updated supernet parameters.

[0088] In one embodiment, the edges of the hypernet are multiple basic operations in the set of basic operations, and the nodes of the hypernet are the hidden state features of the data in the dataset after the basic operations.

[0089] The fine-tuning module is used for:

[0090] Assign the same weight to each basic operation corresponding to the edge, and calculate the output of the node connected to the edge based on the weight;

[0091] Input the dataset into the model to be fine-tuned, adjust the hypernet parameters according to the output of the nodes, and obtain the updated hypernet parameters, which include the updated weights.

[0092] Based on the updated weights, the target basic operation is determined among multiple basic operations;

[0093] In a supernet, removing all basic operations other than the target basic operation yields the target subnet.

[0094] In one embodiment, the basic set of operations includes at least one of the following operations: convolution operation, self-attention connection operation, fully connected operation, and residual connection operation.

[0095] In one embodiment, the apparatus further includes an execution module for:

[0096] Read the data to be processed and input it into the adjusted model;

[0097] In the adjusted model, the target task is executed on the data to be processed, and the processing result corresponding to the target task is obtained.

[0098] In one embodiment, the initialization module is further configured to:

[0099] Acquire training data and use the training data to train a pre-set neural network model to obtain a pre-trained model.

[0100] In one embodiment, an electronic device is provided. This electronic device can be a computer, server, workstation, or other similar device; it can also be a mobile device such as a mobile phone, tablet, or vehicle-mounted mobile terminal; or it can be other devices with program execution capabilities. The internal structure diagram of this electronic device can be as follows: Figure 7 As shown, the electronic device includes a processor, memory, and a network module. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores an operating system, instructions, or code. The internal memory provides an environment for the operation of the operating system and instructions or code in the non-volatile storage media. When the instructions or code are executed by the processor, they implement the functions or steps of a fine-tuning method for a pre-trained model as described above. The network module may include a network interface and / or a wireless network module, allowing the electronic device to communicate with other devices or service platforms. Furthermore, the electronic device may also include a display screen and input devices.

[0101] In one embodiment, an electronic device is provided, including a memory, a processor, and instructions or code stored in the memory and executable on the processor, wherein the processor executes the instructions or code to perform the following steps:

[0102] Obtain a pre-trained model and set a parameter fine-tuning module in each layer of the pre-trained model to obtain the model to be fine-tuned;

[0103] Obtain the dataset corresponding to the target task, and in the model to be fine-tuned, use the differentiable architecture search method to adjust the architecture and parameters of each parameter fine-tuning module according to the dataset to obtain the target parameter fine-tuning module corresponding to the parameter fine-tuning module;

[0104] By combining the pre-trained model and the target parameter fine-tuning module, an adjusted model corresponding to the target task is obtained.

[0105] In one embodiment, a storage medium is provided on which instructions or code are stored, which, when executed by a processor, perform the following steps:

[0106] Obtain a pre-trained model and set a parameter fine-tuning module in each layer of the pre-trained model to obtain the model to be fine-tuned;

[0107] Obtain the dataset corresponding to the target task, and in the model to be fine-tuned, use the differentiable architecture search method to adjust the architecture and parameters of each parameter fine-tuning module according to the dataset to obtain the target parameter fine-tuning module corresponding to the parameter fine-tuning module;

[0108] By combining the pre-trained model and the target parameter fine-tuning module, an adjusted model corresponding to the target task is obtained.

[0109] It should be noted that the functions or steps that the storage medium or electronic device can achieve are described in the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through instructions or code. These instructions or code can be stored in a non-volatile readable storage medium. When executed, the instructions or code can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0111] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0112] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the units or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the units in the system within the embodiment can be distributed throughout the system as described in the embodiment, or they can be modified to reside in one or more systems different from this embodiment. The units in the above-described embodiment can be combined into one unit, or further divided into multiple sub-units.

[0113] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for fine-tuning a pre-trained model, characterized in that, Applied to pathological analysis and medical image analysis, the method includes: A pre-trained model is obtained, and a parameter fine-tuning module is set in each layer of the pre-trained model to obtain a model to be fine-tuned; the pre-trained model is a pre-trained pathological analysis model; the backbone of the pre-trained model has several transformer layers; the parameter fine-tuning module is an adapter module; each layer of the pre-trained model is used to extract features of different degrees from the input sentence or image data; Obtain the dataset corresponding to the target task, and in the model to be fine-tuned, use a differentiable architecture search method to adjust the architecture and parameters of each parameter fine-tuning module according to the dataset to obtain the target parameter fine-tuning module corresponding to the parameter fine-tuning module; the dataset corresponding to the target task includes medical data for pathological analysis and medical image analysis tasks; By combining the pre-trained model and the target parameter fine-tuning module, an adjusted model corresponding to the target task is obtained. Read the data to be processed and input the data to be processed into the adjusted model; in the adjusted model, perform the target task on the data to be processed to obtain the processing result corresponding to the target task; the data to be processed includes tumor tissue slice images; the processing result corresponding to the target task includes tumor type.

2. The method according to claim 1, characterized in that, In the model to be fine-tuned, a differentiable architecture search method is used to adjust the architecture and parameters of each parameter fine-tuning module according to the dataset, to obtain the target parameter fine-tuning module corresponding to the parameter fine-tuning module, including: In each parameter fine-tuning module, a supernet is constructed based on the dataset, and a target subnet is selected in the supernet, which is then used as the architecture of the parameter fine-tuning module. The target parameter fine-tuning module is obtained by adjusting the parameters of the target sub-network using the dataset, wherein the parameters of the target sub-network include at least activation function parameters and encoder parameters.

3. The method according to claim 2, characterized in that, The step of constructing a supernet based on the dataset and selecting a target subnet within the supernet includes: Construct the supernet, wherein the supernet includes multiple layers of network modules, and each layer of the network module includes a set of basic operations; The supernet is trained using the dataset to obtain updated supernet parameters, and the target subnetwork is searched in the supernet based on the updated supernet parameters.

4. The method according to claim 3, characterized in that, The edges of the hypernet are multiple basic operations in the set of basic operations, and the nodes of the hypernet are the hidden state features of the data in the dataset after the basic operations. The step of training the supernet using the dataset to obtain updated supernet parameters, and searching for a target subnetwork within the supernet based on the updated supernet parameters, includes: Assign the same weight to each basic operation corresponding to the edge, and calculate the output of the node connected to the edge based on the weight; The dataset is input into the model to be fine-tuned, and the hypernet parameters are adjusted according to the output of the node to obtain the updated hypernet parameters, wherein the updated hypernet parameters include the updated weights; Based on the updated weights, the target basic operation is determined among the multiple basic operations; In the supernet, other basic operations besides the target basic operation are removed to obtain the target subnet.

5. The method according to claim 4, characterized in that, The basic set of operations includes at least one of the following operations: convolution operation, self-attention connection operation, fully connected operation, and residual connection operation.

6. The method according to claim 1, characterized in that, The acquisition of the pre-trained model includes: Acquire training data and use the training data to train a preset neural network model to obtain the pre-trained model.

7. A fine-tuning device for a pre-trained model, characterized in that, The device, used for pathological analysis and medical image analysis, includes: An initialization module is used to obtain a pre-trained model and set a parameter fine-tuning module in each layer of the pre-trained model to obtain a model to be fine-tuned; the pre-trained model is a pre-trained pathological analysis model; the backbone of the pre-trained model has several transformer layers; the parameter fine-tuning module is an adapter module; each layer of the pre-trained model is used to extract features of different degrees from the input sentence or image data; The fine-tuning module acquires the dataset corresponding to the target task and, in the model to be fine-tuned, uses a differentiable architecture search method to adjust the architecture and parameters of each parameter fine-tuning module according to the dataset, thereby obtaining the target parameter fine-tuning module corresponding to the parameter fine-tuning module; the dataset corresponding to the target task includes medical data for pathological analysis and medical image analysis tasks; The combination module is used to combine the pre-trained model and the target parameter fine-tuning module to obtain the adjusted model corresponding to the target task. An execution module is used to read the data to be processed and input the data to be processed into the adjusted model; in the adjusted model, a target task is performed on the data to be processed to obtain the processing result corresponding to the target task; the data to be processed includes tumor tissue slice images; the processing result corresponding to the target task includes tumor type.

8. A storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the method as described in any one of claims 1 to 6.

9. An electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the program, it implements the method of any one of claims 1 to 6.

Images