Layered self-adaptive freezing fine tuning method and device for multi-modal large model

Abstract

The invention relates to the technical field of artificial intelligence, discloses a hierarchical adaptive freezing fine tuning method and device for a multi-modal large model, and particularly discloses the method comprising the following steps: training a pre-trained multi-modal large model according to a total data set subset to obtain a field expert model; dividing the domain expert model into a plurality of structural layers, and distributing global unfreezing parameter quotas for each structural layer according to contribution degrees; and in each structural layer, performing adaptive importance sorting on each parameter, screening out parameters consistent with the corresponding global unfreezing parameter quota quantity according to an importance sorting sequence to generate a binary mask, and freezing other parameters. And finally, performing sparse training on the domain expert model according to the binary mask. According to the method, the dependence of traditional data playback on old task data or pseudo data generation is overcome, task boundary presetting or manual intervention is not needed, and efficient, lightweight and privacy-safe continuous learning is realized while the generalization ability of the pre-training model is protected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a multi-modal large model hierarchical adaptive freezing fine-tuning method and device. BACKGROUND

[0002] In the prior art, when a model is fine-tuned for a specific task, it may overfit the new task data, resulting in significant degradation of the general knowledge and ability learned in the original pre-training phase. This manifests as a significant decline in performance on the original task or in general scenarios that the model has not participated in fine-tuning while completing the new task. This problem limits the application effect of the model in scenarios that require consideration of multi-task capabilities or dynamic incremental learning.

[0003] To alleviate catastrophic forgetting, existing research mainly revolves around two strategies: one is data replay, the core idea of which is to maintain the model's memory of old knowledge by mixing old task data or generating pseudo data with new task data for training. The other is parameter freezing and regularization, the core idea of which is to reduce the coverage of original knowledge by limiting the update range of model parameters or introducing lightweight modules.

[0004] However, data replay relies on the storage of old task real data or the generation of pseudo data, the former faces privacy leakage and storage cost problems, and the latter is prone to instability in training due to insufficient generated quality. Parameter freezing and regularization methods require pre-setting parameter importance or task boundaries, making it difficult to adapt to dynamic open task flows (such as elastic weight consolidation, which relies on post-computation of parameter importance and cannot respond to task changes in real time). In addition, dynamic architecture expansion (such as task vectors) requires independent parameters for each task, resulting in linear growth of model size and computational overhead, making it difficult to scale up. Therefore, although existing methods can partially alleviate forgetting, they generally rely on limited pre-training data or static task assumptions. SUMMARY

[0005] The present application provides a multi-modal large model hierarchical adaptive freezing fine-tuning method to solve the problem that existing methods can partially alleviate forgetting but generally rely on limited pre-training data or static task assumptions in the prior art.

[0006] Correspondingly, the present application also provides a multi-modal large model hierarchical adaptive freezing fine-tuning device, an electronic device, and a computer-readable storage medium for ensuring the implementation and application of the above method.

[0007] To solve the above technical problems, the present application discloses a multi-modal large model hierarchical adaptive freezing fine-tuning method, which comprises:

[0008] training the pre-trained multi-modal large model according to the subset of the full data set to obtain a domain expert model;

[0009] The domain expert model is divided into multiple structural layers, and a global unfreezing parameter quota is allocated to each structural layer according to the contribution degree;

[0010] In each structural layer, the parameters are adaptively sorted according to importance, and a binary mask is generated by screening parameters consistent with the number of corresponding global unfreezing parameter quotas according to the importance sorting order;

[0011] The domain expert model is sparsely trained according to the binary mask.

[0012] Preferably, before the domain expert model is divided into multiple structural layers and a global unfreezing parameter quota is allocated to each structural layer according to the contribution degree, the method comprises:

[0013] For the target domain task, the absolute change of each parameter is calculated by comparing the parameter difference between the domain expert model and the pre-trained multi-modal large model.

[0014] Preferably, the domain expert model is divided into multiple structural layers, and a global unfreezing parameter quota is allocated to each structural layer according to the contribution degree, comprising:

[0015] The domain expert model is divided into multiple structural layers, and a global unfreezing parameter quota is allocated to each structural layer according to the absolute change of each parameter in each structural layer and the total parameter quantity.

[0016] Preferably, a global unfreezing parameter quota is allocated to each structural layer according to the absolute change of each parameter in each structural layer and the total parameter quantity, comprising:

[0017] Based on the absolute change of each parameter in each structural layer, the average update quantity of each structural layer is calculated.

[0018] Under the global frozen parameter budget of the model, the global unfreezing parameter quota is allocated according to the average update quantity and the total parameter quantity of each structural layer.

[0019] Preferably, under the global frozen parameter budget of the model, the global unfreezing parameter quota is allocated according to the average update quantity and the total parameter quantity of each structural layer, and the formula is:

[0020]

[0021] Wherein, H is the global frozen parameter budget of the model, m l is the average update quantity of the structural layer l, h l is the global unfreezing parameter quota of the structural layer l, and L is the total number of model layers.

[0022] Preferably, in each structural layer, the parameters are adaptively sorted by importance, and the parameters consistent with the corresponding global unfreezing parameter quota number are screened out to generate a binary mask according to the importance sorting order, including:

[0023] All parameters are sorted in descending order of absolute change within each structural layer to obtain the importance sorting order of each parameter.

[0024] According to the importance sorting order, the Top-K function is used to screen out the parameters consistent with the corresponding global unfreezing parameter quota number to generate a binary mask, and the remaining parameters are frozen.

[0025] Preferably, the domain expert model is sparsely trained according to the binary mask, including:

[0026] When fine-tuning the domain expert model, the Hadamard product is used to limit the gradient update to the parameter range marked by the binary mask.

[0027] The application also discloses a multi-modal large model hierarchical adaptive freezing fine-tuning device, which comprises:

[0028] The model supervised fine-tuning module is used to train the pre-trained multi-modal large model according to the subset of the full data set to obtain the domain expert model.

[0029] The hierarchical importance evaluation module is used to divide the domain expert model into a plurality of structural layers, and for each structural layer, the global unfreezing parameter quota is allocated according to the contribution.

[0030] The hierarchical adaptive freezing module is used to, in each structural layer, adaptively sort the parameters by importance, and screen out the parameters consistent with the corresponding global unfreezing parameter quota number to generate a binary mask according to the importance sorting order.

[0031] The model parameter updating module is used to sparsely train the domain expert model according to the binary mask.

[0032] The application also discloses an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements one or more methods described in the application when executing the program.

[0033] The application also discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement one or more methods described in the application.

[0034] In the present application, a pre-trained multi-modal large model is trained according to a subset of a full data set to obtain a domain expert model, the domain expert model is divided into multiple structure layers, and for each structure layer, a global unfreezing parameter quota is allocated according to the contribution degree; in each structure layer, each parameter is adaptively sorted according to importance, and parameters consistent with the number of the corresponding global unfreezing parameter quota are screened out according to the importance sorting order to generate a binary mask, and other parameters are frozen. Through dynamic parameter sensitivity analysis and hierarchical adaptive freezing strategy, the dependence on old task data or generated pseudo data of traditional data playback is overcome, the parameter update amount is used to quantify the key sub-network and generate a binary mask, and the problems of privacy leakage and generated noise are avoided. Moreover, by dynamically evaluating the importance of parameters, there is no need to preset the task boundary or manual intervention, and the adaptability of open tasks is significantly improved; finally, the domain expert model is sparsified according to the binary mask, ensuring that the model size and training time are constant, reducing the overhead, and protecting the generalization ability of the pre-trained model while realizing efficient, lightweight and privacy-safe continuous learning.

[0035] Additional aspects and advantages of the present application will be described in the following description part, which will become apparent from the following description or be understood by those skilled in the art through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0036] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0037] Figure 1 A flowchart of the multi-modal large model hierarchical adaptive freezing fine-tuning method provided by the embodiments of the present application is shown in FIG. 1;

[0038] Figure 2 A detailed flowchart of the hierarchical adaptive freezing fine-tuning provided by the embodiments of the present application is shown in FIG. 2;

[0039] Figure 3 A structure schematic diagram of the multi-modal large model hierarchical adaptive freezing fine-tuning device provided by the embodiments of the present application is shown in FIG. 3;

[0040] Figure 4 A structure schematic diagram of the electronic device provided by the embodiments of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0041] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, in which the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as a limitation on the present application.

[0042] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is further understood that the terms "comprise" and "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It is further understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements can be present. In addition, the use of "connection" or "coupling" herein also includes wireless connection or wireless coupling. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0043] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless specifically so defined, as herein.

[0044] The scheme provided by the embodiments of the present application can be executed by any electronic device, which can be a terminal device or a server. The server can be a physical server, a server cluster composed of multiple physical servers, a distributed system, or a cloud server providing cloud computing services. The terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the present application. For the technical problems existing in the prior art, the multi-modal large model hierarchical adaptive freezing fine-tuning method and device provided by the present application aims to solve at least one of the technical problems in the prior art.

[0045] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail in the specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0046] Embodiments of the present application aim to effectively alleviate the knowledge coverage and forgetting phenomenon caused by parameter updating under the premise of ensuring the adaptability of the model to new tasks, thereby improving the comprehensive performance and stability of the model in the cross-task and continuous learning scene by optimizing the fine-tuning strategy and training mechanism. Based on this, the present application provides a multi-modal large model hierarchical adaptive freezing fine-tuning method, which can be executed by any electronic device, and can be executed on a server side or a terminal device.

[0047] As shown in Figure 1 , the method can include the following steps:

[0048] Step 101, training the pre-trained multi-modal large model according to the subset of the full data set to obtain a domain expert model.

[0049] As shown in Figure 2 , the full data set of the new domain is denoted as The full data set is divided into a subset α, and the pre-trained multi-modal large model is supervised and fine-tuned using the subset to obtain a domain expert model ε t .

[0050] Step 102, dividing the domain expert model into multiple structural layers, and for each structural layer, allocating a global unfreezing parameter quota according to the contribution degree.

[0051] Embodiments of the present application divide the domain expert model and the pre-trained multi-modal large model into L structural layers, denoted as layer 1, layer 2, layer 3, …, layer L, then analyze the contribution degree of each layer, and automatically allocate more global unfreezing parameter quotas. The contribution degree refers to the contribution degree of all parameters in the structural layer to the structural layer, which is measured by the total parameter quantity and the parameter change quantity. Each structural layer has different contribution degrees, and the higher the contribution degree, the more global unfreezing parameter quotas are allocated.

[0052] Step 103, in each structural layer, adaptively sorting the parameters according to importance, and filtering out parameters consistent with the number of corresponding global unfreezing parameter quotas according to the importance sorting order to generate a binary mask.

[0053] Step 104, sparse training the domain expert model according to the binary mask.

[0054] In the embodiments of the present application, the pre-trained multi-modal large model is trained according to the subset of the full data set to obtain a domain expert model, the domain expert model is divided into multiple structural layers, and the global unfreezing parameter quota is allocated according to the contribution degree for each structural layer; in each structural layer, the parameters are adaptively sorted according to importance, and the binary mask is generated by screening the parameters consistent with the number of the corresponding global unfreezing parameter quota according to the importance sorting order, and the other parameters are frozen. Through dynamic parameter sensitivity analysis and hierarchical adaptive freezing strategy, the dependence on old task data or generated pseudo data of traditional data playback is overcome, the parameter update amount is used to quantify the key sub-network and generate a binary mask, and the problems of privacy leakage and generated noise are avoided. Moreover, by dynamically evaluating the importance of the parameters, the task boundary is preset or manual intervention is not required, and the adaptability of open tasks is significantly improved; finally, the domain expert model is sparsified according to the binary mask, so that the model size and training time are constant, and the computing overhead is reduced, while the generalization ability of the pre-trained model is protected, the efficient, lightweight and privacy-safe continuous learning is realized.

[0055] In an optional embodiment, before the domain expert model is divided into multiple structural layers and the global unfreezing parameter quota is allocated according to the contribution degree for each structural layer, the method comprises:

[0056] For the target domain task, the absolute change amount of each parameter is calculated by comparing the parameter difference between the domain expert model and the pre-trained multi-modal large model.

[0057] In the embodiments of the present application, the pre-trained multi-modal large model is supervised and fine-tuned by the subset of the full data set, and in the process of supervised fine-tuning, the parameter update sensitivity is quantitatively analyzed, the parameter difference between the domain expert model and the pre-trained multi-modal large model is compared, and the absolute change amount of each parameter is calculated

[0058]

[0059] wherein θ i is the i-th parameter of the pre-trained multi-modal large model, is the i-th parameter of the domain expert model.

[0060] In the embodiments of the present application, only the subset of the full data set is used to supervise and fine-tune the pre-trained multi-modal large model, which can realize adaptation to a new field with extremely low parameter amount, and significantly reduce the computing overhead.

[0061] In an optional embodiment, for the target domain task, the domain expert model is divided into multiple structural layers, and the global unfreezing parameter quota is allocated according to the contribution degree for each structural layer, comprising:

[0062] The domain expert model is divided into multiple structure layers, and the global unfreezing parameter quota is proportionally allocated to each structure layer based on the absolute change amount and total parameter amount of each parameter in each structure layer.

[0063] In an optional embodiment, the global unfreezing parameter quota is proportionally allocated to each structure layer based on the absolute change amount and total parameter amount of each parameter in each structure layer, including:

[0064] The average update amount of each structure layer is calculated based on the absolute change amount of each parameter in the structure layer.

[0065] Under the global frozen parameter budget of the model, the global unfreezing parameter quota is proportionally allocated according to the average update amount and total parameter amount of each structure layer.

[0066] The formula for layering statistical average update amount is as follows:

[0067]

[0068] Wherein, m l is the average update amount of structure layer l.

[0069] In an optional embodiment, under the global frozen parameter budget of the model, the global unfreezing parameter quota is proportionally allocated according to the average update amount and total parameter amount of each structure layer, and the formula is:

[0070]

[0071] Wherein, H is the global frozen parameter budget of the model, h l is the global unfreezing parameter quota of structure layer l, and L is the total number of model layers.

[0072] In an optional embodiment, in each structure layer, the parameters are adaptively sorted according to importance, and the binary mask is generated by screening parameters consistent with the number of corresponding global unfreezing parameter quota according to the importance sorting order Including:

[0073] All parameters in each structure layer are sorted in descending order according to the absolute change amount to obtain the importance sorting order of each parameter;

[0074] According to the importance sorting order, the binary mask is generated by screening high-sensitive parameters consistent with the number of corresponding global unfreezing parameter quota h l through the Top-K function , and the remaining parameters are frozen. The specific implementation formula is as follows:

[0075]

[0076] Wherein, represents w l real vector space, represents the index of the top K largest elements.

[0077] Through the above hierarchical adaptive freezing strategy, the adjustable parameter budget is dynamically allocated, and the parameters are automatically frozen, which can ensure that the high-sensitive layer obtains more adjustable parameters, while avoiding manual presetting of task boundaries.

[0078] In an optional embodiment, the field expert model is trained by sparsification according to a binary mask, including:

[0079] When fine-tuning the field expert model, the Hadamard product (⊙) is used to strictly limit the gradient update to the parameter range marked by the binary mask The parameter range marked by the binary mask:

[0080]

[0081] wherein, represents a loss function. In the embodiments of the present application, sparsification training is achieved through dynamic gradient mask update, which strictly limits the gradient update to the parameter range marked by the binary mask The key parameters are updated, and the generalization ability of the pre-trained multi-modal large model is maximally preserved.

[0082] The embodiments of the present application dynamically identify and update the key parameters in the field expert model, realize efficient field adaptation and generalization ability protection of the multi-modal large model. Specifically, the above hierarchical adaptive freezing fine-tuning method proposed in the embodiments of the present application does not need to rely on old task data storage or generate pseudo data, does not need to increase the training overhead, through an inter-layer adaptive importance sorting mechanism, the contribution of each layer to the new task is automatically analyzed, the parameter importance sorting is dynamically generated and sparsification update is implemented, and the preset task boundary or manual intervention is avoided, thereby realizing efficient, lightweight and privacy-safe continuous learning in an open task scenario, realizing the optimization balance of field performance and model efficiency. The method in the embodiments of the present application reduces the computing overhead by 70% compared with the dynamic architecture expansion method (such as task vector).

[0083] To verify the effectiveness of the multi-modal large model hierarchical adaptive frozen fine-tuning method proposed in the embodiments of the present application in reducing catastrophic forgetting, the performance of the full-parameter fine-tuning method, the low-rank adaptive fine-tuning method and the hierarchical adaptive frozen fine-tuning method proposed in the embodiments of the present application is evaluated on a series of data sets, including DocVQA, ChartQA, AI2D, TextVQA, OCRBench, InfoVQA, Seed2Plus, and MMBench data sets, and the results are shown in Table 1. Compared with the full-parameter fine-tuning, the method in the embodiments of the present application has improved on all data sets, and the performance of the method in the embodiments of the present application is better than the baseline on AI2D, TextVQA, InfoVQA, and Seed2Plus data sets.

[0084] Table 1 Comparison of catastrophic forgetting degrees of various training methods

[0085]

[0086]

[0087] In the embodiments of the present application, an OCR data set is constructed based on the training data in the OCR field, and an InternVL3-8B model is used as a base model. It is found through experiments that if the InternVL3-8B model is directly fine-tuned on the above-mentioned OCR data set, catastrophic forgetting phenomenon will occur in the model, which is specifically manifested as a significant performance decline on general visual question answering tasks and multi-modal reasoning benchmark tests, and the model is difficult to maintain the original processing capability of general tasks. To solve this problem, a multi-modal large model hierarchical adaptive frozen fine-tuning method is proposed in the embodiments of the present application. The method trains a pre-trained multi-modal large model according to a subset of the OCR data set to obtain a domain expert model. The domain expert model is divided into multiple structural layers, and for each structural layer, a global unfreezing parameter quota is allocated according to the contribution. In each structural layer, the parameters are adaptively sorted according to importance, and the parameters consistent with the number of global unfreezing parameter quota are selected according to the importance sorting order to generate a binary mask. The domain expert model is sparsely trained according to the binary mask. This method effectively alleviates the catastrophic forgetting effect while adapting to the new OCR domain task.

[0088] In the embodiments of the present application, the performance of the full-parameter fine-tuning method, the low-rank adaptive fine-tuning method and the hierarchical adaptive frozen fine-tuning method proposed in the embodiments of the present application is evaluated on the OCR data set, and the results are shown in Table 2.

[0089] Table 2 Comparison of various training methods on the OCR data set

[0090] Method OCR dataset (edit distance down) Baseline 0.28 Full-parametric fine-tuning 0.37 Low-rank adaptive fine-tuning 0.27 Hierarchical adaptive frozen fine-tuning 0.27

[0091] The results show that the method proposed in the embodiments of the present application can prevent catastrophic forgetting after supervised fine-tuning, and its performance on the OCR dataset is also improved. In addition, on some datasets such as TextVQA, InfoVQA and Seed2plus, the method proposed in the embodiments of the present application is superior to the base model, which verifies the effectiveness of the intelligent protection training strategy in the embodiments of the present application.

[0092] The method in the embodiments of the present application can be extended to the field of model lightweight, realizing high-performance compression through sensitivity-driven parameter pruning; or adapting to cross-domain migration of multi-modal models, such as separating modal-specific parameters in visual-language tasks.

[0093] Based on the same principle as the method provided in the embodiments of the present application, the embodiments of the present application also provide a multi-modal large model hierarchical adaptive freezing fine-tuning device, as shown in Figure 3 The device comprises:

[0094] The model supervised fine-tuning module 301 is configured to train the pre-trained multi-modal large model according to the subset of the full data set to obtain a domain expert model;

[0095] The hierarchical importance evaluation module 302 is configured to divide the domain expert model into a plurality of structure layers, and allocate a global unfreezing parameter quota according to the contribution degree for each structure layer;

[0096] The hierarchical adaptive freezing module 303 is configured to, in each structure layer, perform adaptive importance sorting on each parameter, and filter out parameters consistent with the number of corresponding global unfreezing parameter quotas according to the importance sorting order to generate a binary mask;

[0097] The model parameter updating module 304 is configured to perform sparse training on the domain expert model according to the binary mask.

[0098] In the embodiments of the present application, the pre-trained multi-modal large model is trained according to the subset of the full data set to obtain a domain expert model, the domain expert model is divided into multiple structure layers, and for each structure layer, the global unfreezing parameter quota is allocated according to the contribution degree; in each structure layer, the parameters are adaptively sorted according to importance, and the parameters consistent with the number of the corresponding global unfreezing parameter quota are screened out according to the importance sorting order to generate a binary mask, and other parameters are frozen. Through dynamic parameter sensitivity analysis and hierarchical adaptive freezing strategy, the dependence on old task data or generated pseudo data of traditional data playback is overcome, the parameter update amount is used to quantify the key sub-network and generate a binary mask, and the problems of privacy leakage and generated noise are avoided. Moreover, by dynamically evaluating the importance of parameters, there is no need to preset the task boundary or manual intervention, and the adaptability of open tasks is significantly improved; finally, the domain expert model is sparsified according to the binary mask, so as to ensure the constant model size and training time, reduce the computational overhead, and realize efficient, lightweight and privacy-safe continuous learning while protecting the generalization ability of the pre-trained model.

[0099] The multi-modal large model hierarchical adaptive freezing fine-tuning device provided by the embodiments of the present application can realize Figure 1 and The processes realized in the method embodiments of Figure 2 will not be described here again to avoid repetition.

[0100] The multi-modal large model hierarchical adaptive freezing fine-tuning device of the embodiments of the present application can execute the multi-modal large model hierarchical adaptive freezing fine-tuning method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module and unit in the multi-modal large model hierarchical adaptive freezing fine-tuning device of the embodiments of the present application are corresponding to the steps in the multi-modal large model hierarchical adaptive freezing fine-tuning method of the embodiments of the present application. For detailed descriptions of the functions of each module of the multi-modal large model hierarchical adaptive freezing fine-tuning device, please refer to the descriptions of the corresponding multi-modal large model hierarchical adaptive freezing fine-tuning method in the foregoing, which will not be described here again.

[0101] Based on the same principle as the method shown in the embodiments of the present application, the embodiments of the present application also provide an electronic device, which can include but is not limited to: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the hierarchical adaptive freezing fine-tuning method of the multi-modal large model shown in any optional embodiment of the present application by calling the computer program. Compared with the prior art, the hierarchical adaptive freezing fine-tuning method of the multi-modal large model provided in the present application trains the pre-trained multi-modal large model according to the subset of the full data set, obtains the domain expert model, divides the domain expert model into multiple structure layers, and allocates global unfreezing parameter quota according to the contribution degree for each structure layer; in each structure layer, the parameters are adaptively sorted according to importance, and the parameters consistent with the number of global unfreezing parameter quota are screened out according to the importance sorting order to generate a binary mask, and other parameters are frozen. Through dynamic parameter sensitivity analysis and hierarchical adaptive freezing strategy, the dependence on old task data or generated pseudo data by traditional data playback is overcome, the parameter update amount is used to quantify the key sub-network and generate a binary mask, and the problems of privacy leakage and generated noise are avoided. Moreover, by dynamically evaluating the importance of parameters, there is no need to preset the task boundary or manual intervention, and the adaptability of open tasks is significantly improved; finally, the domain expert model is sparsified according to the binary mask, ensuring the constant model size and training time, reducing the computational overhead, and realizing efficient, lightweight and privacy-safe continuous learning while protecting the generalization ability of the pre-trained model.

[0102] In an optional embodiment, an electronic device is also provided, as shown in Figure 4 Figure 4 The electronic device 400 shown in the optional embodiment can be a server, which includes a processor 401 and a memory 403. The processor 401 and the memory 403 are connected, such as through a bus 402. Optionally, the electronic device 400 can also include a transceiver 404. It should be noted that in actual applications, the transceiver 404 is not limited to one, and the structure of the electronic device 400 does not constitute a limitation on the embodiments of the present application.

[0103] ​The processor 401 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It can implement or execute the various exemplary logical blocks, modules and circuits described in connection with the disclosure. The processor 401 can also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0104] The bus 402 can include a path for transmitting information between the above-mentioned components. The bus 402 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 402 can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 4 Only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0105] The memory 403 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, an optical disk storage (including a compact disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited thereto.

[0106] The memory 403 is configured to store application program codes for implementing the solutions of the present application, and the processor 401 is configured to control the execution of the application program codes.

[0107] The electronic device includes, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a car terminal (for example, a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. Figure 4 The electronic device shown is merely an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0108] The server provided in the present application can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like, but is not limited thereto. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the present application.

[0109] The present application provides a computer readable storage medium having a computer program stored thereon, which, when executed on a computer, enables the computer to perform the corresponding content in the foregoing method embodiments.

[0110] It should be understood that, although each step in the flowchart of the accompanying drawings is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other orders. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or sub-steps or stages of other steps.

[0111] It should be noted that the computer readable storage medium in the above application can also be a computer readable signal medium or a combination of a computer readable storage medium and a computer readable storage medium. The computer readable storage medium may, for example, be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In this application, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0112] The above computer readable medium can be contained in the above electronic device; or can exist separately and not be assembled into the electronic device.

[0113] The above computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0114] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the multi-modal large model hierarchical adaptive fine-tuning method and device provided in the various optional implementation manners.

[0115] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0116] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0117] The modules involved in the embodiments of the present application can be implemented in a software manner or in a hardware manner. Among them, the name of the module does not constitute a limitation of the module itself in some cases. For example, the model supervision fine-tuning module can also be described as "a model supervision fine-tuning module for training a pre-trained multi-modal large model according to a subset of a full data set to obtain a domain expert model".

[0118] The above description is only the preferred embodiment of the present application and the explanation of the technical principles. It should be understood by those skilled in the art that the disclosed scope of the present application is not limited to the technical scheme composed of the specific combination of the above technical features, and should also cover other technical schemes formed by the combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the technical scheme formed by the mutual replacement of the above features and the technical features disclosed in the present application (but not limited to) with similar functions.

Claims

1. A multimodal large model hierarchical adaptive freezing fine-tuning method, characterized by: The method comprises: Train the pre-trained multimodal large model based on a subset of the full dataset to obtain a domain expert model; Divide the domain expert model into multiple structural layers, and allocate a global unfreezing parameter quota to each structural layer according to its contribution; In each structural layer, each parameter is adaptively ranked by importance, and the parameters that are consistent with the corresponding global unfreeze parameter quota are selected according to the importance ranking order to generate a binary mask; The domain expert model is subjected to sparse training according to the binary mask.

2. The multimodal large model hierarchical adaptive freezing fine-tuning method according to claim 1 is characterized in that: Before dividing the domain expert model into multiple structural layers and allocating a global unfreezing parameter quota to each structural layer according to its contribution, the method includes: For the target domain task, the absolute change of each parameter is calculated by comparing the parameter differences between the domain expert model and the pre-trained multimodal large model.

3. The multimodal large model hierarchical adaptive freezing fine-tuning method according to claim 2 is characterized in that: The domain expert model is divided into multiple structural layers, and a global unfreezing parameter quota is allocated to each structural layer according to its contribution, including: The domain expert model is divided into multiple structural layers, and a global unfreezing parameter quota is proportionally allocated to each structural layer based on the absolute change amount of each parameter in each structural layer and the total parameter amount.

4. The multimodal large model hierarchical adaptive freezing fine-tuning method according to claim 3 is characterized in that: The global unfreezing parameter quota is allocated to each structural layer in proportion based on the absolute change of each parameter in each structural layer and the total parameter amount, including: Based on the absolute changes of each parameter in each structural layer, the average update amount of each structural layer is calculated; Under the global frozen parameter budget of the model, the global unfrozen parameter quota is allocated proportionally according to the average update amount and total parameter amount of each structural layer.

5. The multimodal large model hierarchical adaptive freezing fine-tuning method according to claim 4 is characterized in that: Under the model global frozen parameter budget, the formula for allocating the global unfrozen parameter quota proportionally according to the average update amount and total parameter amount of each structural layer is: Among them, H is the global frozen parameter budget of the model, m l is the average update amount of the structure layer l, h l is the global unfreezing parameter quota of structure layer l, and L is the total number of model layers.

6. The multimodal large model hierarchical adaptive freezing fine-tuning method according to claim 1 is characterized in that: In each structural layer, each parameter is adaptively sorted by importance, and parameters that are consistent with the corresponding global unfreeze parameter quota are screened out in the order of importance to generate a binary mask, including: All parameters in each structural layer are sorted in descending order according to the absolute change amount to obtain the importance ranking of each parameter; According to the importance ranking order, the Top-K function is used to filter out parameters that are consistent with the corresponding global unfreeze parameter quota to generate a binary mask, and the remaining parameters are frozen.

7. The multimodal large model hierarchical adaptive freezing fine-tuning method according to claim 1 is characterized in that: The performing sparse training on the domain expert model according to the binary mask includes: When fine-tuning the domain expert model, the Hadamard product is used to limit the gradient update to the parameter range marked by the binary mask.

8. A multi-modal large model hierarchical adaptive freezing fine-tuning device, characterized in that: The device comprises: The model supervision fine-tuning module is used to train the pre-trained multimodal large model based on a subset of the full dataset to obtain a domain expert model; The hierarchical importance evaluation module is used to divide the domain expert model into multiple structural layers and allocate global unfreezing parameter quotas to each structural layer according to its contribution; The layered adaptive freezing module is used to adaptively sort the importance of each parameter in each structural layer, and select the parameters that are consistent with the corresponding global unfreeze parameter quota in the order of importance to generate a binary mask; A model parameter updating module is used to perform sparse training on the domain expert model according to the binary mask.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the method according to any one of claims 1 to 7 is implemented when the processor executes the program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images