Multi-task processing method and system for large language model fused with LoRA

Abstract

The embodiment of the invention relates to the technical field of large language modules, and discloses a multi-task processing method of a large language model fused with LoRA, which comprises the following steps: receiving task processing information input by a user, and performing feature extraction on the task processing information to determine each scene task feature; performing data matching with a pre-LoRA module parameter set based on the scene task features to obtain a LoRA module parameter set associated with the scene task parameters, and performing parameter updating operation on the large language model according to the LoRA module parameter set; and inputting the scene task features into the large language model subjected to the parameter updating operation for identification to obtain corresponding identification result information, and outputting the corresponding identification result information. According to the scheme of the invention, by fusing the LoRA module and the routing mechanism, the adaptive selection and optimization capability for specific tasks is realized, and the performance of the model in processing diversified tasks is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a multi-task processing method and system for a large language model integrating LoRA. Background Art

[0002] At present, large language models play a vital role in accelerating the progress of natural language processing technology, providing a general and task-independent basic platform for a variety of applications. However, the inherent diversity of data in a specific field poses a significant challenge to training a general basic model. In view of this, the development of large models designed to solve specific field problems has increased rapidly, such as SQL-PaLM, BloombergGPT, ChatLaw, pdfGPT, etc., to cope with the diverse customization needs in the real world. These models with diverse customization functions can effectively solve a wide range of user problems, and each functional module can be optimized independently. Domain specialization technology is a key factor in realizing the innovative application of large language models in a variety of application scenarios. In order to master sufficient domain knowledge without sacrificing basic functions, and then improve the accuracy of the overall model in responding to specific field problems has become a technical problem that technicians in this field need to solve urgently. Summary of the invention

[0003] In response to the above-mentioned defects, an embodiment of the present invention discloses a multi-task processing method of a large language model integrating LoRA, which realizes adaptive selection and optimization capabilities for specific tasks by integrating the LoRA module and the routing mechanism, thereby improving the performance of the model in processing diversified tasks.

[0004] The first aspect of the embodiment of the present invention discloses a multi-task processing method of a large language model integrating LoRA, including:

[0005] Receive task processing information input by a user, and perform feature extraction on the task processing information to determine the features of each scenario task;

[0006] Based on the scenario task characteristics, data matching is performed with a pre-set LoRA module parameter set to obtain a LoRA module parameter group associated with the scenario task parameters, and a parameter update operation is performed on the large language model according to the LoRA module parameter group; wherein the LoRA module parameter set includes a plurality of LoRA module parameter groups, and each LoRA module parameter group corresponds one-to-one to a specific task scenario;

[0007] The scene task features are input into the large language model after the parameter update operation for recognition to obtain corresponding recognition result information, and the corresponding recognition result information is output.

[0008] As an optional implementation, in the first aspect of the embodiment of the present invention, the large language model includes multiple Transformer layers, and multiple LoRA modules are simultaneously incorporated into each Transformer layer, and each LoRA module has an independent parameter group; a routing component is embedded before each Transformer layer, and the routing component is used to match the corresponding LoRA module based on the model input data;

[0009] The large language model is constructed by the following steps:

[0010] Obtain model training sets based on data from various fields, each of which is associated with a task type parameter;

[0011] Adding a low-rank adapter to the pre-trained large language model, and training the pre-trained large language model based on model training sets constructed based on data from various fields until the pre-trained large language model meets corresponding conditions;

[0012] Obtain fine-tuning module parameters for each field of data, and store the fine-tuning module parameters in the corresponding LoRA module;

[0013] The parameter updating operation of the large language model according to the LoRA module parameter group includes:

[0014] The LoRA module parameters of the scene domain are merged with the model parameters of the large language model to obtain the target large language model of the corresponding user demand domain.

[0015] As an optional implementation, in the first aspect of the embodiment of the present invention, the model training sets constructed based on the data in various fields are used to train the pre-trained large language models respectively, including:

[0016] In the process of integrating the LoRA module into the pre-trained large language model, a trainable routing parameter Ψ is introduced, and the trainable routing parameter Ψ and the parameter θ of each LoRA module are converted into i They are all used as model parameters of the pre-trained large language model for joint optimization training.

[0017] As an optional implementation, in the first aspect of the embodiment of the present invention, the Transformer layer of the large language model includes a word embedding projection matrix P W , query the projection matrix P Q , key projection matrix P K , value projection matrix P V , the intermediate layer projection matrix P M , output layer projection matrix P O , the weight matrix W in the multilayer perceptron w;The architecture of the Transformer layer includes:

[0018] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O ;

[0019]

[0020] Where Q, K, and V represent query, key, and value vectors respectively; W O is the output transformation matrix; is the corresponding linear transformation matrix; d k is the dimension of the key vector, used to scale the dot product; MultiHead(Q,K,V) is the final output;

[0021] The Transformer-based parameter fine-tuning model uses the low-rank parameter increment matrix A and the low-rank parameter increment matrix B and the transformation matrix T to update the pre-trained parameter matrix ΔW, where the pre-trained parameter matrix is fixed, rank r<<k or d,

[0022] During the training process, the transformation matrix T performs task-specific adaptive linear transformations on the low-rank parameter increment matrix A and the low-rank parameter increment matrix B, wherein the adaptive linear transformations include scaling, rotation, translation, reflection, tilt, and projection; the transformation matrix T includes one or more of a spatial harmonic integral matrix, an intrinsic correlation feature matrix, a composite transformation coupling matrix, and a dual transformation superposition matrix.

[0023] As an optional implementation, in the first aspect of the embodiment of the present invention, the pre-trained large language model is constructed by the following steps:

[0024] The final loss of the large language model system integrating LoRA is the sum of the language modeling loss and the mixed expert model routing loss. The final loss function is: Among them, η is a tuning parameter used to balance the impact of the two loss weights on the model training speed and convergence; represents the cross entropy loss based on expert classification; Represents the language modeling loss, which is expressed as follows:

[0025] The language modeling task is a common goal of large language models for pre-training decoders, where x represents a sequence of tokens {x1, x2, …, x n}, x i is the preceding token x in the given sequence<i Target marker for posterior prediction;

[0026] A Transformer-based parameter fine-tuning model is adopted, and the transformation matrix T is used to optimize the LoRA module parameters to enhance the model's adaptability. The specific definition is as follows:

[0027] W=W0+T·router(W r ,A1B2,…,A N B N )

[0028] Among them, W0 represents the initial weight matrix of the large language model; W r is the router parameter; A i and B i is the parameter of the i-th LoRA module, which is adjusted through the transformation matrix T to adapt to specific task requirements.

[0029] As an optional implementation, in the first aspect of the embodiment of the present invention, the pre-trained large language model is optimized by the following steps:

[0030] Acquire domain metadata, and classify the input task processing information using the domain metadata, wherein the domain metadata includes task category data and context tag data;

[0031] A sequence-level routing strategy is used to route the domain metadata to the corresponding LoRA module. During training, each Transformer layer uses a different router to distribute the training data labeled with the corresponding domain. During the training process, a multi-layer perceptron is used as a router.

[0032] Multiple LoRA modules are placed in parallel with W0, and different LoRA modules are assigned to handle tasks in different fields. The LoRA module updates the weight using the following formula: W0+router(W r ,W1,…,W N )

[0033] =W0+router(W r ,A1B1,…,A N B N )

[0034] in, W0 represents the initial attention and feedforward weight matrix of the basic large language model, whose parameters are fixed; the parameters of router(·) are trainable; W r is the trainable parameter of the router, implemented by the linear layer; A i B i Defines LoRA module Mi , A i and B i are the parameters of the i-th LoRA module, This module is repeated multiple times within each Transformer layer, so that the data of each task type can be processed by the most suitable expert module to reduce the trainable parameters to adapt to different domain tasks.

[0035] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the multi-task processing method further includes:

[0036] Divide the data in each field into user permissions to determine the access rights of different users; the access rights are used to indicate the LoRA modules with corresponding field permissions;

[0037] Obtain domain permission information associated with the user; if it is found that the user does not have permission to obtain the application of specific domain parameters, the user's domain permission information is matched with each domain in the LoRA module for similarity to obtain the LoRA module with the highest similarity to the user's domain permission information;

[0038] The parameters in the large language model are updated using the LoRA module or the general domain model parameters that have the highest similarity to the user's domain authority information;

[0039] After outputting the corresponding recognition result information, the method further includes:

[0040] The user is reminded of information based on the parameters in the LoRA module used.

[0041] A second aspect of an embodiment of the present invention discloses a multi-task processing system integrating a large language model of LoRA, including:

[0042] Receiving module: used to receive task processing information input by the user, and perform feature extraction on the task processing information to determine the task features and context information of each scenario;

[0043] Routing module: used to perform data matching with a pre-set LoRA module parameter set based on the scenario task features and context information to obtain a LoRA module parameter group associated with the scenario task parameters, and perform parameter update operation on the large language model according to the LoRA module parameter group; wherein the LoRA module parameter set includes multiple LoRA module parameter groups, and each LoRA module parameter group corresponds to a specific task scenario one by one;

[0044] Recognition module: used to input the scene task features and context information into the large language model that has undergone parameter update operation for recognition to obtain corresponding recognition result information, and output the corresponding recognition result information.

[0045] The third aspect of an embodiment of the present invention discloses an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the multi-task processing method of a large language model integrating LoRA disclosed in the first aspect of the embodiment of the present invention.

[0046] A fourth aspect of an embodiment of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the multi-task processing method of a large language model integrating LoRA disclosed in the first aspect of an embodiment of the present invention.

[0047] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0048] The multi-task processing method of the large language model integrating LoRA in the embodiment of the present invention uses a predefined LoRA module parameter set, and the method can quickly adapt the parameters of the large language model for different task scenarios. LoRA technology fine-tunes the pre-trained model through a low-rank matrix. Compared with full parameter fine-tuning, this method has less computational complexity and faster parameter updates, thereby achieving efficient adaptability of the model to different tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 It is a flowchart of a multi-task processing method of a large language model integrating LoRA disclosed in an embodiment of the present invention;

[0051] Figure 2 This is an architecture diagram of the integration of LoRA and routing mechanism disclosed in an embodiment of the present invention

[0052] Figure 3 It is an architecture diagram of a Transformer-based parameter fine-tuning model and four transformation matrices disclosed in an embodiment of the present invention;

[0053] Figure 4 This is a sequence-level routing strategy architecture diagram integrating LoRA disclosed in an embodiment of the present invention

[0054] Figure 5 It is a structural diagram of a multi-task processing system integrating a large language model of LoRA provided by an embodiment of the present invention;

[0055] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] It should be noted that the terms "first", "second", "third", "fourth", etc. in the specification and claims of the present invention are used to distinguish different objects rather than to describe a specific order. The terms "including" and "having" in the embodiments of the present invention and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0058] Domain specialization techniques are key factors in achieving innovative applications of large language models in a variety of application scenarios. In order to master sufficient domain knowledge without sacrificing basic functions, adapter-based fine-tuning methods, such as adapters and LoRA, retain domain-related knowledge by introducing a limited number of domain-specific parameters. This approach does not require fine-tuning all parameters of the pre-trained model, effectively reducing the training cost. In order to achieve the diverse customization capabilities of the model, directly mixing data from multiple domains and fine-tuning by adding only one LoRA module is a simple and efficient method. In addition, there are some two-stage methods, such as Pfeiffer and Huang et al., which implicitly integrate the knowledge of multiple task adapters by adding AdapterFusion layers and element-wise LoRA combination. The MoE mixed expert model handles different types of input data by introducing multiple experts, while Jiang et al. aim to comprehensively evaluate the output of the comprehensive large model.

[0059] Domain specialization of large language models can be divided into three methods: external enhancement, prompt generation, and model fine-tuning. Current large language models have billions of parameters and face the challenge of catastrophic forgetting. Adapter-based fine-tuning methods, such as Adapter and LoRA, address this by training multiple domain experts on different task data in advance. As a parameter-efficient fine-tuning method, LoRA facilitates the adaptation of large language models using smaller external modules. Therefore, LoRA fine-tuning provides a resource-efficient technique for quickly adapting large language models to new tasks with limited training data.

[0060] Mixture of Experts MoE is an ensemble approach that is typically presented as a set of submodules or “experts” that specialize in processing different types of input data. Each expert is controlled by a router and selectively activated based on the type of input data. This technique has shown excellent performance in other fields such as computer vision, speech recognition, and multimodal applications. Shen et al. have also demonstrated that MoE combined with instruction adjustment can improve the performance of specific tasks. The fusion of multi-task capabilities is usually achieved through two-stage learning or end-to-end multi-task learning methods. Two-stage methods rely on maintaining specialized routing models to support multiple task models, and their overall performance is limited by a single LoRA model. End-to-end methods introduce new parameter layers or perform implicit parameter fusion, such as the fusion parameter layer trained by Pfeiffer et al. to combine the information stored in multiple task adapters. Huang et al. integrated multiple LoRA modules into a unified module through a set of learnable weights. Although these methods can achieve a certain degree of generalization ability by combining parameters trained on multiple tasks, this fusion method may reduce the performance of the original task. Zhao et al. also pointed out that it is extremely challenging to effectively learn all the expertise in different fields in one large model. Based on this, the embodiment of the present invention discloses a multi-task processing method, system, electronic device and storage medium of a large language model integrating LoRA, which can quickly adapt the parameters of the large language model for different task scenarios through a predefined LoRA module parameter set. LoRA technology fine-tunes the pre-trained model through a low-rank matrix. Compared with full parameter fine-tuning, this method has less computational complexity and faster parameter updates, thereby achieving efficient adaptability of the model to different tasks.

[0061] Embodiment 1

[0062] See also Figure 1 , Figure 1It is a flow chart of the multi-task processing method of the large language model integrating LoRA disclosed in the embodiment of the present invention. Among them, the execution subject of the method described in the embodiment of the present invention is an execution subject composed of software and / or hardware, and the execution subject can receive relevant information by wired or / and wireless means, and can send certain instructions. Of course, it can also have certain processing functions and storage functions. The execution subject can control multiple devices, such as a remote physical server or cloud server and related software, or it can be a local host or server and related software that performs related operations on a device placed somewhere. In some scenarios, multiple storage devices can also be controlled, and the storage devices can be placed in the same place or different places as the devices. For example Figures 1 to 4 As shown, the multi-task processing method based on the large language model integrating LoRA includes the following steps:

[0063] S101: receiving task processing information input by a user, and performing feature extraction on the task processing information to determine the features of each scenario task;

[0064] S102: Based on the scenario task feature, data matching is performed with a pre-set LoRA module parameter set to obtain a LoRA module parameter group associated with the scenario task parameter, and a parameter update operation is performed on the large language model according to the LoRA module parameter group; wherein the LoRA module parameter set includes a plurality of LoRA module parameter groups, and each LoRA module parameter group corresponds one-to-one to a specific task scenario;

[0065] S103: Inputting the scene task features into the large language model after the parameter update operation for recognition to obtain corresponding recognition result information, and outputting the corresponding recognition result information.

[0066] The embodiment of the present invention describes a multi-task processing method of a large language model integrating LoRA (Low-Rank Adaptation) technology, and its technical effects are mainly reflected in the following aspects:

[0067] Efficient task adaptability: Through the pre-defined LoRA module parameter set, this method can quickly adapt the parameters of the large language model for different task scenarios. LoRA technology fine-tunes the pre-trained model through a low-rank matrix. Compared with full parameter fine-tuning, this method has less computational complexity and faster parameter updates, thereby achieving efficient adaptability of the model to different tasks. Resource saving: Since LoRA only modifies a small part of the model's parameters (i.e., LoRA module parameters) instead of all the parameters of the entire model, this method significantly reduces the demand for storage and computing resources. This is particularly important for deploying large language models in resource-constrained environments such as edge devices. Maintaining model stability: By modifying only the LoRA module parameters, most of the knowledge of the original pre-trained model is retained, which helps maintain the stability and generalization ability of the model when processing other tasks. At the same time, it also reduces the risk of overfitting that may be caused by full parameter fine-tuning. Multi-task processing capability: This method achieves the multi-task processing capability of the large language model by matching different LoRA module parameter groups to cope with diverse task scenarios. This flexibility enables the model to easily switch between different fields and tasks, improving the practicality and application value of the model. Improve recognition accuracy: By optimizing parameters for specific task scenarios, this method can improve the recognition accuracy and performance of large language models on corresponding tasks. This targeted optimization makes the model more accurate and efficient when processing specific tasks. Simplify model deployment and maintenance: Since the LoRA module parameter set can be updated and maintained independently of the original pre-trained model, the model deployment and maintenance process is simplified. When you need to adapt to new tasks or optimize existing tasks, you only need to update the corresponding LoRA module parameters without retraining the entire model.

[0068] In summary, the large language model multi-task processing method integrating LoRA has shown significant technical effects in improving task adaptability, saving resources, maintaining model stability, enhancing multi-task processing capabilities, improving recognition accuracy, and simplifying model deployment and maintenance.

[0069] More preferably, the large language model includes multiple Transformer layers, and multiple LoRA modules are simultaneously incorporated into each Transformer layer, and each LoRA module has an independent parameter group; a routing component is embedded before each Transformer layer, and the routing component is used to match the corresponding LoRA module based on the model input data;

[0070] The large language model is constructed by the following steps:

[0071] S1021: Obtain model training sets constructed based on data from various fields, each model training set being associated with a task type parameter;

[0072] S1022: adding a low-rank adapter to the pre-trained large language model, and training the pre-trained large language model respectively based on the model training sets constructed based on the data in various fields until the pre-trained large language model meets the corresponding conditions;

[0073] S1023: Obtain fine-tuning module parameters for data in various fields, and store the fine-tuning module parameters in corresponding LoRA modules;

[0074] The parameter updating operation of the large language model according to the LoRA module parameter group includes:

[0075] The LoRA module parameters of the scene domain are merged with the model parameters of the large language model to obtain the target large language model of the corresponding user demand domain.

[0076] In view of the limitations and challenges of existing solutions, the present invention proposes a large language model system integrating LoRA. The present invention realizes adaptive selection and optimization capabilities for specific tasks by integrating LoRA modules and routing mechanisms, adopts a parameter fine-tuning model based on the Transformer architecture, and introduces four transformation strategies to optimize the parameter configuration of the LoRA module, thereby significantly improving the efficiency of the model in processing multiple tasks. In addition, by constructing and optimizing the comprehensive loss function, the present invention strengthens the adaptability and generalization capabilities of the model. Specifically, the present invention enables the model to be adaptively adjusted according to task requirements by integrating the LoRA module and the routing mechanism, and fine-tunes the LoRA module parameters through the carefully designed Transformer architecture fine-tuning method and various transformation strategies, which not only improves the performance of the model in processing diversified tasks, but also enhances the adaptability and generalization of the model through the optimization of the final loss function. The present invention effectively solves the catastrophic forgetting problem and the challenges of multi-task dynamic tuning, improves the adaptability and generalization of the algorithm, and realizes multi-task tuning of large language models.

[0077] The embodiment of the present invention can achieve more precise control of model parameters by incorporating multiple LoRA modules into each Transformer layer and assigning each module an independent parameter group. This layered and modular design enables the model to be optimized for features at different levels, thereby further improving the performance of the model when processing complex tasks.

[0078] The routing component embedded before each Transformer layer can dynamically match the corresponding LoRA module according to the model input data. This flexible routing mechanism enables the model to adaptively select the most appropriate LoRA module for processing according to different task scenarios and data characteristics, improving the adaptability and flexibility of the model.

[0079] Since each LoRA module is trained and optimized for specific domain data, when multiple LoRA modules work simultaneously, the large language model can handle diverse tasks more effectively. This design enhances the multi-task processing capability of the model, enabling the model to handle multiple domains or tasks simultaneously, improving the practicality and application value of the model.

[0080] In the parameter update operation, by merging the obtained scenario domain LoRA module parameters with the model parameters of the large language model, the target large language model of the corresponding user demand domain can be quickly constructed. This parameter update method is not only computationally efficient, but also can retain the stability and generalization ability of the model when processing other tasks.

[0081] Since the design of LoRA modules and routing components are modular and extensible, when new domain data or tasks need to be introduced, new LoRA modules and routing rules can be easily added without large-scale reconstruction of the entire model. This design improves the scalability and maintainability of the model.

[0082] A method that combines low-rank adaptive LoRA technology with a routing mechanism is constructed. This method can improve the performance and flexibility of the model in a multi-task learning environment by adaptively selecting modules optimized for specific tasks. Specifically, the optimal LoRA module parameters are first obtained on each scene task data, and then the routing mechanism is used to integrate N LoRA modules into a shared large language model. Multiple LoRA modules are simultaneously incorporated into each Transformer layer, and before each Transformer layer, a routing mechanism is inserted to guide the selection of different LoRA experts.

[0083] First, for N different task types N LoRA modules were designed and trained separately Each LoRA module R i For its corresponding task M i Optimize to achieve the best performance on this specific task; train the corresponding LoRA module on each task-related dataset to obtain the module parameter set θ that is optimized for a specific task scenario i ;

[0084] A routing mechanism is used, which can dynamically select the most suitable LoRA module for processing according to the characteristics or context information of the input data; specifically, the system embeds a routing component before each Transformer layer, which is responsible for selecting the appropriate LoRA module based on the input data and incorporating it into the calculation process of the large language model; this design allows the system to flexibly adjust its internal structure according to the specific needs of the input data, thereby improving processing efficiency and accuracy of the results;

[0085] In the process of integrating the LoRA modules into the shared large language model, trainable router parameters Ψ are introduced, which are related to the parameters θ of each LoRA module. i The purpose of optimizing together is to fine-tune the routing decision process to ensure that the most suitable expert module can be selected for each task type; through this parameterized routing strategy, the system can achieve a significant improvement in multi-task processing capabilities while maintaining efficient autoregressive language modeling;

[0086] More preferably, the model training sets constructed based on data from various fields are used to train the pre-trained large language model respectively, including:

[0087] In the process of integrating the LoRA module into the pre-trained large language model, a trainable routing parameter Ψ is introduced, and the trainable routing parameter Ψ and the parameter θ of each LoRA module are converted into i They are all used as model parameters of the pre-trained large language model for joint optimization training.

[0088] The trainable routing parameters of the embodiment of the present invention can be automatically adjusted according to the feedback during the model training process to more accurately match the optimal LoRA module in different task scenarios. This adaptive routing mechanism further enhances the adaptability of the model to different tasks, making the model more accurate and efficient when processing diverse tasks.

[0089] Using routing parameters and LoRA module parameters as model parameters for joint optimization training can ensure that the model learns how to optimally select LoRA modules and how to adjust LoRA module parameters to adapt to specific tasks during the training process. This joint optimization strategy helps improve the overall training effect of the model, so that the model has better performance and generalization ability after training.

[0090] Traditional multi-task processing models often require manual specification of which tasks should use which parameters or modules during training. After the introduction of trainable routing parameters, this process can be completed automatically, reducing the need for human intervention and reducing the complexity and cost of model training.

[0091] Since both routing parameters and LoRA module parameters are trainable, when new tasks or domain data need to be introduced, it is only necessary to expand the corresponding LoRA module and routing rules and continue the joint optimization training. This design enables the model to easily cope with new task requirements and improves the scalability of the model.

[0092] By jointly optimizing the training routing parameters and LoRA module parameters, the model can better learn the commonalities and differences between different tasks, thereby improving the model's generalization ability when dealing with unseen tasks or domains. This improvement in generalization ability is of great significance for model deployment and performance in practical applications.

[0093] More preferably, the Transformer layer of the large language model includes a word embedding projection matrix P W , query the projection matrix P Q , key projection matrix P K , value projection matrix P V , the intermediate layer projection matrix P M , output layer projection matrix P O , the weight matrix W in the multilayer perceptron w ;The architecture of the Transformer layer includes:

[0094] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O ;

[0095]

[0096] Where Q, K, and V represent query, key, and value vectors respectively; W O is the output transformation matrix; is the corresponding linear transformation matrix; d k is the dimension of the key vector, used to scale the dot product; MultiHead(Q,K,V) is the final output;

[0097] The Transformer-based parameter fine-tuning model uses the low-rank parameter increment matrix A and the low-rank parameter increment matrix B and the transformation matrix T to update the pre-trained parameter matrix ΔW, where the pre-trained parameter matrix is fixed, rank r<<k or d,

[0098] During the training process, the transformation matrix T performs task-specific adaptive linear transformations on the low-rank parameter increment matrix A and the low-rank parameter increment matrix B, wherein the adaptive linear transformations include scaling, rotation, translation, reflection, tilt, and projection; the transformation matrix T includes one or more of a spatial harmonic integral matrix, an intrinsic correlation feature matrix, a composite transformation coupling matrix, and a dual transformation superposition matrix.

[0099] Specifically, we designed a parameter fine-tuning model based on the Transformer architecture, and improved the performance of the model in the field of multi-task natural language processing by introducing transformation strategies and parameterization methods. We also achieved efficient and flexible adjustment of model parameters by introducing linear transformations and change matrices with specific structures.

[0100] The Transformer architecture used contains multiple dense parameter matrices, each of which carries specific semantic information and processing capabilities; it contains multiple modules, including the word embedding projection matrix P W , query the projection matrix P Q , key projection matrix P K , value projection matrix P V , the intermediate layer projection matrix P M , output layer projection matrix P O , the weight matrix W in the multilayer perceptron w ;

[0101] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O

[0102] Where Q, K, and V represent query, key, and value vectors respectively; W O is the output transformation matrix;

[0103]

[0104] in, is the corresponding linear transformation matrix;

[0105]

[0106] Among them, d k is the dimension of the key vector, which is used to scale the dot product to prevent the softmax function from entering the saturation region;

[0107] Transformer-based parameter fine-tuning model: The parameter matrix of a specific task is linearly transformed using the change matrix T, and its spatial geometric structure is changed through linear transformation to generate a new matrix feature pattern, thereby improving the performance of the fine-tuned model after training; specifically, the Transformer-based parameter fine-tuning model uses a low-rank parameter increment matrix and And the transformation matrix The product B(T)A is used to approximate the pre-trained parameter matrix The update of the pre-trained parameter matrix is fixed, where the rank r<<k or d;

[0108] During training, T performs task-specific adaptive linear transformations on A and B, including scaling, rotation, translation, reflection, tilt, and projection. T contains four different structures: spatial harmonic integral matrix, intrinsic correlation feature matrix, composite transformation coupling matrix, and dual transformation superposition matrix.

[0109] Spatial harmonic integral matrix: It integrates a variety of different spatial feature transformations. By introducing the transformation matrix C, the A and B parameter matrices are subjected to spatial rotation, scaling, translation, shearing and other linear transformations according to the specific task to form a comprehensive feature representation. This integration not only enhances the adaptability of the model to different tasks, but also provides richer expression capabilities for adjusting model parameters, so that the model can adapt to and optimize the processing effect of specific tasks. Its forward propagation expression in the model is as follows:

[0110] h=(W0+ΔW)x=W0x+ΔWx=W0x+(BCA)x

[0111] in, x represents the input vector; h represents the output vector; Since r<<k or d, the number of parameters in the C transformation matrix is ​​very small, and the added overhead is minimal; the A and C matrices are initialized using random Gaussian initialization, while the B matrix is ​​initialized to a zero matrix;

[0112] Intrinsic correlation feature matrix: By introducing the transformation matrix C and its transpose C T , after multiplication, a semi-positive definite matrix is ​​obtained, which can simulate the covariance matrix that captures the intrinsic correlation structure in the feature space; this positive semi-definite matrix applies linear transformations such as rotation and scaling to the A and B parameter matrices, highlighting the task-specific features in the feature space, thereby enhancing the representation ability of the model by increasing the distinction between patterns; its forward propagation expression in the model is as follows:

[0113] h=(W0+ΔW)x=W0x+ΔWx=W0x+B(CC T)Ax

[0114] in, represents the transpose of matrix C;

[0115] Composite transformation coupling matrix: By introducing two different transformation matrices C and D and forming a composite transformation coupling matrix through matrix multiplication, the expressiveness of the parameter increment matrix is ​​enhanced, thereby cultivating the model's adaptability to complex tasks; its forward propagation expression in the model is as follows:

[0116] h=(W0+ΔW)x=W0x+ΔWx=W0x+B(CD)Ax

[0117] in,

[0118] Dual transformation superposition matrix: The dual transformation superposition matrix is ​​formed by the addition operation of matrices C and D to promote complex interactions between model parameters; this matrix allows for detailed scaling and modulation of model features, thereby providing the model with enhanced expressiveness and flexibility to meet complex task requirements; its forward propagation expression in the model is as follows:

[0119] h=(W0+ΔW)x=W0x+ΔWx=W0x+B(D+C)Ax

[0120] in, Allowing more complex interactions between A and B enhances the expressiveness and flexibility of the model.

[0121] The embodiment of the present invention introduces low-rank parameter increment matrices A and B, and transformation matrix T, so that the model can be efficiently fine-tuned without changing the original pre-trained parameter matrix W_0. This method significantly reduces the number of parameters that need to be updated, reduces computational complexity and storage requirements, and maintains the performance of the model when processing new tasks.

[0122] The transformation matrix T performs task-specific adaptive linear transformations on the low-rank parameter increment matrices A and B, including scaling, rotation, translation, reflection, tilt, and projection. This adaptive linear transformation enables the model to flexibly adjust its internal parameters according to the requirements of different tasks, thereby enhancing the adaptability and generalization ability of the model.

[0123] The transformation matrix T can include various types such as spatial harmonic integral matrix, intrinsic correlation feature matrix, composite transformation coupling matrix and dual transformation superposition matrix. These different types of transformation matrices provide the model with rich transformation capabilities, enabling the model to more accurately capture the subtle differences between tasks and further improve the performance of the model.

[0124] Since the pre-trained parameter matrix W_0 is fixed and only the low-rank parameter increment matrix A, B and the transformation matrix T are updated, this method helps to maintain the stability and consistency of the model when processing other tasks. At the same time, since the rank of the low-rank parameter increment matrix is ​​much smaller than the dimension of the original parameter matrix, the noise and interference introduced during the update process are also relatively small, further ensuring the stability of the model.

[0125] By combining the powerful feature extraction capabilities of the Transformer layer and the efficient fine-tuning method based on low-rank parameter increments, the large language model can show higher performance and accuracy when processing diverse tasks. This performance improvement is not only reflected in a single task, but also in multi-task processing scenarios, enabling the model to better meet the needs of practical applications.

[0126] More preferably, the pre-trained large language model is constructed by the following steps:

[0127] The final loss of the large language model system integrating LoRA is the sum of the language modeling loss and the mixed expert model routing loss. The final loss function is: Among them, η is a tuning parameter used to balance the impact of the two loss weights on the model training speed and convergence; represents the cross entropy loss based on expert classification; Represents the language modeling loss, which is expressed as follows:

[0128] The language modeling task is a common goal of large language models for pre-training decoders, where x represents a sequence of tokens {x1, x2, …, x n}, x i is the preceding token x in the given sequence <i Target marker for posterior prediction;

[0129] A Transformer-based parameter fine-tuning model is adopted, and the transformation matrix T is used to optimize the LoRA module parameters to enhance the model's adaptability. The specific definition is as follows:

[0130] W=W0+T·router(W r ,A1B2,…,A N B N )

[0131] Among them, W0 represents the initial weight matrix of the large language model; W r is the router parameter; A i and B i is the parameter of the i-th LoRA module, which is adjusted through the transformation matrix T to adapt to specific task requirements.

[0132] In the embodiment of the present invention, the LoRA module is introduced and the parameters of the LoRA module are optimized by using the transformation matrix T during the fine-tuning process, so that the model can adapt to different task requirements more flexibly. This adaptive capability enables the model to maintain high performance and accuracy when processing diverse tasks, while reducing the risk of overfitting.

[0133] Combining the language modeling loss L_LM(x) and the cross entropy loss L_clsfy based on expert classification, and balancing the weights of the two by adjusting the parameter λ, helps the model focus on both the accuracy of language modeling and the accuracy of expert classification during training. This balancing strategy helps improve the overall performance of the model and accelerates the convergence of the model.

[0134] A Transformer-based parameter fine-tuning model is adopted, and the LoRA module parameters are adjusted using the transformation matrix T to achieve fine control of the model parameters. This method not only reduces the number of parameters that need to be updated and reduces the computational complexity, but also improves the performance of the model when processing new tasks while maintaining the stability of the model. By integrating the LoRA module, optimizing the loss function, and using the transformation matrix for parameter fine-tuning, the pre-trained large language model can show higher performance and accuracy when processing diverse tasks. This performance improvement is not only reflected in a single task, but can also play an important role in multi-task processing scenarios to meet the complex needs of practical applications.

[0135] By introducing the cross entropy loss L_clsfy for expert classification, the model can learn the associations and differences between different experts (or LoRA modules) during training. This information helps enhance the interpretability of the model, allowing developers to more clearly understand the model's behavior and decision-making process.

[0136] The embodiment of the present invention is adjusted through the role of the transformation matrix T to adapt to specific task requirements; the optimization strategy aims to dynamically adjust the parameters of the LoRA module so that the model can provide customized processing capabilities for different task types.

[0137] More preferably, the pre-trained large language model is optimized by the following steps:

[0138] Acquire domain metadata, and classify the input task processing information using the domain metadata, wherein the domain metadata includes task category data and context tag data;

[0139] A sequence-level routing strategy is used to route the domain metadata to the corresponding LoRA module. During training, each Transformer layer uses a different router to distribute the training data labeled with the corresponding domain. During the training process, a multi-layer perceptron is used as a router.

[0140] Multiple LoRA modules are placed in parallel with W0, and different LoRA modules are assigned to handle tasks in different fields. The LoRA module updates the weight using the following formula: W0+router(W r ,W1,…,W N )

[0141] =W0+router(W r ,A1B1,…,A N B N )

[0142] in, W0 represents the initial attention and feedforward weight matrix of the basic large language model, whose parameters are fixed; the parameters of router(·) are trainable; W r is the trainable parameter of the router, implemented by the linear layer; A i B i Defines LoRA module M i , A i and B i are the parameters of the i-th LoRA module, This module is repeated multiple times within each Transformer layer, so that the data of each task type can be processed by the most suitable expert module to reduce the trainable parameters to adapt to different domain tasks.

[0143] Specifically, a sequence-level routing strategy is designed to dynamically route data to specific low-rank adaptive expert modules by utilizing domain metadata to improve the ability and efficiency of large language models in handling multi-task natural language processing problems; Specifically, a sequence-level routing strategy is adopted to classify input data using domain metadata, and the most appropriate LoRA expert module is selected for processing based on the classification results; The router embedded before each Transformer layer is responsible for executing this dynamic selection process, determining which expert module the data should be routed to based on the characteristics or contextual information of the input sequence;

[0144] A sequence-level routing strategy is adopted to route data to LoRA experts using domain metadata. Although all token sequences traverse the weight matrix in the Transformer layer of the large language model, each Transformer layer uses a different router to distribute training data labeled with the corresponding domain during training. The final routing loss forces accurate data-to-expert allocation for each data type, which can improve generation efficiency and context consistency. After training with an appropriate amount of balanced data, a robust router is obtained, and multi-task training enhances the generalization ability of the original task. The expert selection in the reasoning process is optimized by voting or selecting the last expert, ensuring smooth switching and high adaptability of the model between different tasks.

[0145] A LoRA expert is explicitly designed for each domain, and LoRA updates the weights using the following formula:

[0146] W0+router(W r ,W1,…,W N )

[0147] =W0+router(W r ,A1B1,…,A N B N )

[0148] in, W0 represents the initial attention and feedforward weight matrix of the basic large language model, whose parameters are fixed; the parameters of router(·) are trainable; W r is the trainable parameter of the router, implemented by the linear layer; A i B i Defines the LoRA module A i and B i are the parameters of the i-th LoRA module, This module is repeated multiple times within each Transformer layer, so that the data of each task type can be processed by the most suitable expert module to reduce the trainable parameters to adapt to different domain tasks;

[0149] Multiple Experts are placed in parallel with W0, and different experts are assigned to handle tasks in different fields, which helps to avoid interference between tasks; a router is added to each Transformer layer to select the most suitable expert, and each router is implemented as a two-layer multi-layer perceptron MLP, which will not significantly increase the number of training parameters; the training goal of the router is to optimize routing decisions to maximize the overall model performance.

[0150] By acquiring domain metadata and classifying the input task processing information, the model can more accurately identify task requirements in different fields. Subsequently, a sequence-level routing strategy is used to route the domain metadata to the corresponding LoRA module, ensuring that tasks in each field can be processed by the most suitable expert module. This design significantly enhances the adaptability of the model to different fields and improves the accuracy and efficiency of task processing.

[0151] Each Transformer layer uses different routers to distribute training data labeled with the corresponding field, and multiple LoRA modules are placed in parallel with the base model weight W_0. This parallel processing mechanism enables the model to handle tasks in multiple fields at the same time, making full use of computing resources. At the same time, since the LoRA module has fewer parameters, the overall number of parameters of the model has not increased significantly, maintaining a high computational efficiency.

[0152] Since the LoRA module is independent of the base model weight W_0, when new domain tasks need to be processed, new LoRA modules can be easily added and trained without retraining the entire model. This flexible model scalability enables the model to easily cope with changing task requirements.

[0153] During the training process, a multi-layer perceptron is used as a router, and the parameters of the router are optimized to achieve effective scheduling of the LoRA module. This scheduling strategy helps to accelerate the training process of the model and promote the rapid convergence of the model. At the same time, since the parameters of the LoRA module are relatively small and updated frequently, they can capture the characteristic changes of the domain task more quickly, thereby further improving the training efficiency and convergence speed of the model.

[0154] More preferably, the multi-task processing method further includes:

[0155] Divide the data in each field into user permissions to determine the access rights of different users; the access rights are used to indicate the LoRA modules with corresponding field permissions;

[0156] Obtain domain permission information associated with the user; if it is found that the user does not have permission to obtain the application of specific domain parameters, the user's domain permission information is matched with each domain in the LoRA module for similarity to obtain the LoRA module with the highest similarity to the user's domain permission information;

[0157] The parameters in the large language model are updated using the LoRA module or the general domain model parameters that have the highest similarity to the user's domain authority information;

[0158] After outputting the corresponding recognition result information, the method further includes:

[0159] The user is reminded of information based on the parameters in the LoRA module used.

[0160] The embodiment of the present invention divides the user rights of data in each field, ensuring that only users with corresponding field rights can access and use the LoRA module in that field. This permission control mechanism effectively prevents unauthorized users from accessing sensitive field data or performing unauthorized operations, thereby improving the security of the system.

[0161] When a user does not have permission to apply parameters to a specific domain, the system can automatically match the user's domain permission information with the various domains in the LoRA module to find the LoRA module that is most similar to the user's permission information. This mechanism allows users to obtain services close to their needs even without direct permission, thereby improving user experience.

[0162] The parameters in the large language model are updated using the LoRA module or general domain model parameters that are most similar to the user's domain authority information, ensuring that the model can fully utilize the domain knowledge that is most relevant to the user when processing user requests. This flexible model application method enables the model to understand user intent more accurately and give responses that are more in line with user expectations.

[0163] After outputting the recognition result information, the system can also remind the user based on the parameters in the adopted LoRA module. This information reminder mechanism can not only help users understand the domain knowledge and parameter settings used in the model processing process, but also enhance the interactivity between users and models and improve user satisfaction. This method helps protect the security of user data and privacy by limiting the user's access to data in specific fields and providing alternative services through similarity matching when necessary. This mechanism reduces the risk of sensitive data leakage and provides users with a safer usage environment.

[0164] Embodiment 2

[0165] See also Figure 5 , Figure 5 Schematic diagram of the structure of the multi-task processing system of the large language model integrating LoRA disclosed in the embodiment of the present invention. Figure 5 As shown, the multi-task processing system integrating the large language model of LoRA may include:

[0166] Receiving module 21: used to receive task processing information input by a user, and perform feature extraction on the task processing information to determine the task features and context information of each scenario;

[0167] Routing module 22: used to perform data matching with a pre-set LoRA module parameter set based on the scenario task features and context information to obtain a LoRA module parameter group associated with the scenario task parameters, and perform parameter update operation on the large language model according to the LoRA module parameter group; wherein the LoRA module parameter set includes multiple LoRA module parameter groups, and each LoRA module parameter group corresponds to a specific task scenario one by one;

[0168] Recognition module 23: used for inputting the scene task features and context information into the large language model after the parameter update operation to perform recognition to obtain corresponding recognition result information, and output the corresponding recognition result information.

[0169] The multi-task processing method of the large language model integrating LoRA in the embodiment of the present invention uses a predefined LoRA module parameter set, and the method can quickly adapt the parameters of the large language model for different task scenarios. LoRA technology fine-tunes the pre-trained model through a low-rank matrix. Compared with full parameter fine-tuning, this method has less computational complexity and faster parameter updates, thereby achieving efficient adaptability of the model to different tasks.

[0170] Embodiment 3

[0171] See also Figure 6 , Figure 6 Schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. The electronic device may be a computer, a server, etc. Of course, in certain circumstances, it may also be a smart device such as a mobile phone, a tablet computer, a monitoring terminal, and an image acquisition device with processing functions. Figure 6 As shown, the electronic device may include:

[0172] A memory 510 storing executable program codes;

[0173] a processor 520 coupled to the memory 510;

[0174] The processor 520 calls the executable program code stored in the memory 510 to execute some or all of the steps in the multi-task processing method of the large language model integrating LoRA in the first embodiment.

[0175] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute part or all of the steps in the multi-task processing method of a large language model integrating LoRA in Embodiment 1.

[0176] An embodiment of the present invention also discloses a computer program product, wherein when the computer program product runs on a computer, the computer executes part or all of the steps in the multi-task processing method of the large language model integrating LoRA in embodiment one.

[0177] An embodiment of the present invention also discloses an application publishing platform, wherein the application publishing platform is used to publish a computer program product, wherein when the computer program product runs on a computer, the computer executes part or all of the steps in the multi-tasking processing method of the large language model integrating LoRA in embodiment one.

[0178] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the processes does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed over multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0180] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0181] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several requests for a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to perform some or all of the steps of the method described in each embodiment of the present invention.

[0182] In the embodiments provided by the present invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0183] A person of ordinary skill in the art can understand that some or all of the steps in the various methods of the embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0184] The above is a detailed introduction to the multi-task processing method, system, electronic device and storage medium of the large language model integrating LoRA disclosed in the embodiments of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A multi-task processing method integrating a large language model of LoRA, characterized in that: include: Receive task processing information input by a user, and perform feature extraction on the task processing information to determine the features of each scenario task; Based on the scenario task characteristics, data matching is performed with a pre-set LoRA module parameter set to obtain a LoRA module parameter group associated with the scenario task parameters, and a parameter update operation is performed on the large language model according to the LoRA module parameter group; wherein the LoRA module parameter set includes a plurality of LoRA module parameter groups, and each LoRA module parameter group corresponds one-to-one to a specific task scenario; The scene task features are input into the large language model after the parameter update operation for recognition to obtain corresponding recognition result information, and the corresponding recognition result information is output.

2. The multi-task processing method of the large language model integrating LoRA as claimed in claim 1, characterized in that: The large language model includes multiple Transformer layers, and multiple LoRA modules are simultaneously incorporated into each Transformer layer, and each LoRA module has an independent parameter group; a routing component is embedded before each Transformer layer, and the routing component is used to match the corresponding LoRA module based on the model input data; The large language model is constructed by the following steps: Obtain model training sets based on data from various fields, each of which is associated with a task type parameter; Adding a low-rank adapter to the pre-trained large language model, and training the pre-trained large language model based on model training sets constructed based on data from various fields until the pre-trained large language model meets corresponding conditions; Obtain fine-tuning module parameters for each field of data, and store the fine-tuning module parameters in the corresponding LoRA module; The parameter updating operation of the large language model according to the LoRA module parameter group includes: The LoRA module parameters of the scene domain are merged with the model parameters of the large language model to obtain the target large language model of the corresponding user demand domain.

3. The multi-task processing method of the large language model integrating LoRA as claimed in claim 2, characterized in that: The model training sets constructed based on data from various fields are used to train the pre-trained large language models respectively, including: In the process of integrating the LoRA module into the pre-trained large language model, a trainable routing parameter Ψ is introduced, and the trainable routing parameter Ψ and the parameter θ of each LoRA module are converted into i They are all used as model parameters of the pre-trained large language model for joint optimization training.

4. The multi-task processing method of the large language model integrating LoRA as claimed in claim 2, characterized in that: The Transformer layer of the large language model includes a word embedding projection matrix P W , query the projection matrix P Q , key projection matrix P K , value projection matrix P V , the intermediate layer projection matrix P M , output layer projection matrix P O , the weight matrix W in the multilayer perceptron w ;The architecture of the Transformer layer includes: MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O ; Where Q, K, and V represent query, key, and value vectors respectively; W O is the output transformation matrix; is the corresponding linear transformation matrix; d k is the dimension of the key vector, used to scale the dot product; MultiHead(Q,K,V) is the final output; The Transformer-based parameter fine-tuning model uses the low-rank parameter increment matrix A and the low-rank parameter increment matrix B and the transformation matrix T to update the pre-trained parameter matrix ΔW, where the pre-trained parameter matrix is fixed, rank r<<k or d, During the training process, the transformation matrix T performs task-specific adaptive linear transformations on the low-rank parameter increment matrix A and the low-rank parameter increment matrix B, wherein the adaptive linear transformations include scaling, rotation, translation, reflection, tilt, and projection; the transformation matrix T includes one or more of a spatial harmonic integral matrix, an intrinsic correlation feature matrix, a composite transformation coupling matrix, and a dual transformation superposition matrix.

5. The multi-task processing method of the large language model integrating LoRA as claimed in claim 4, characterized in that: The pre-trained large language model is constructed by the following steps: The final loss of the large language model system integrating LoRA is the sum of the language modeling loss and the mixed expert model routing loss. The final loss function is: Among them, η is a tuning parameter used to balance the impact of the two loss weights on the model training speed and convergence; represents the cross entropy loss based on expert classification; Represents the language modeling loss, which is expressed as follows: The language modeling task is a common goal of large language models for pre-training decoders, where x represents a sequence of tokens {x1, x2, ..., x n }, x i is the preceding token x in the given sequence ＜i Target marker for posterior prediction; A Transformer-based parameter fine-tuning model is adopted, and the transformation matrix T is used to optimize the LoRA module parameters to enhance the model's adaptability. The specific definition is as follows: W=W0+T·router(W r ,A1B2,…,A N B N ) Among them, W0 represents the initial weight matrix of the large language model; W r is the router parameter; A i and B i is the parameter of the i-th LoRA module, which is adjusted through the transformation matrix T to adapt to specific task requirements.

6. The multi-task processing method of the large language model integrating LoRA as claimed in claim 5, characterized in that: The pre-trained large language model is optimized through the following steps: Acquire domain metadata, and classify the input task processing information using the domain metadata, wherein the domain metadata includes task category data and context tag data; A sequence-level routing strategy is used to route the domain metadata to the corresponding LoRA module. During training, each Transformer layer uses a different router to distribute training data labeled with the corresponding domain; During the training process, a multilayer perceptron was used as a router; Multiple LoRA modules are placed in parallel with W0, and different LoRA modules are assigned to handle tasks in different fields. The LoRA module updates the weight using the following formula: W0+router(W r , W1,…,W N )=W0+router(W r , A1B1,…,A N B N ) in, W0 represents the initial attention and feedforward weight matrix of the basic large language model, whose parameters are fixed; the parameters of router(·) are trainable; W r is the trainable parameter of the router, implemented by the linear layer; A i B i Defines LoRA module M i , A i and B i are the parameters of the i-th LoRA module, This module is repeated multiple times within each Transformer layer, so that the data of each task type can be processed by the most suitable expert module to reduce the trainable parameters to adapt to different domain tasks.

7. The multi-task processing method of the large language model integrating LoRA as claimed in claim 1, characterized in that: The multi-tasking method further includes: Divide the data in each field into user permissions to determine the access rights of different users; the access rights are used to indicate the LoRA modules with corresponding field permissions; Obtain domain permission information associated with the user; if it is found that the user does not have permission to obtain the application of specific domain parameters, the user's domain permission information is matched with each domain in the LoRA module for similarity to obtain the LoRA module with the highest similarity to the user's domain permission information; The parameters in the large language model are updated using the LoRA module or the general domain model parameters that have the highest similarity to the user's domain authority information; After outputting the corresponding recognition result information, the method further includes: The user is reminded of information based on the parameters in the LoRA module used.

8. A multi-task processing system integrating a large language model of LoRA, characterized in that: include: Receiving module: used to receive task processing information input by the user, and perform feature extraction on the task processing information to determine the task features and context information of each scenario; Routing module: used to perform data matching with a pre-set LoRA module parameter set based on the scenario task features and context information to obtain a LoRA module parameter group associated with the scenario task parameters, and perform parameter update operation on the large language model according to the LoRA module parameter group; wherein the LoRA module parameter set includes multiple LoRA module parameter groups, and each LoRA module parameter group corresponds to a specific task scenario one by one; Recognition module: used to input the scene task features and context information into the large language model that has undergone parameter update operation for recognition to obtain corresponding recognition result information, and output the corresponding recognition result information.

9. An electronic device, characterized in that: include: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the multi-tasking processing method of the large language model integrating LoRA as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program enables a computer to execute the multi-task processing method of a large language model integrating LoRA as described in any one of claims 1 to 7.

Images