Multi-task oriented large language model construction method and device, equipment and medium

Abstract

The application relates to the technical field of model construction, and discloses a multi-task-oriented large language model construction method, device, equipment and medium. The method comprises the following steps: constructing a low-rank matrix corresponding to each preset task type respectively, and configuring a target key vector for each low-rank matrix; acquiring task data corresponding to each preset task type, and extracting a semantic feature vector of the task data; calculating the cosine similarity of the semantic feature vector and each target key vector, and mapping the cosine similarity to a routing score when the corresponding low-rank matrix processes the task data; selecting at least one target low-rank matrix from the multiple low-rank matrices according to the routing score; and performing model representation learning on a preset large language model according to the target low-rank matrix, so as to obtain a multi-task large language model for parallel processing of multiple different task types. The application can be applied to a financial technology, medical health and other business system platform, and can improve the efficiency of task parallel processing.

Description

Technical Field

[0001] This invention relates to the field of model building technology, and in particular to a method, apparatus, device and medium for building large language models for multiple tasks. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) in natural language processing, how to enable them to efficiently handle multiple tasks (such as text classification, named entity recognition, question answering, and summarization) has become a research hotspot. Currently, most mainstream multi-task solutions employ joint fine-tuning across multiple tasks, i.e., training model parameters uniformly on all task data. However, this approach requires updating all or most of the model's parameters simultaneously. Gradient directions for different tasks often conflict, easily leading to negative transfer, especially when task boundaries are ambiguous or task data have similar semantics; parameter sharing can actually exacerbate the error rate in routing. Furthermore, a single large language model struggles to process multiple task types in parallel, often requiring independent training of multiple models in practice, resulting in significantly increased resource consumption and poor knowledge consistency.

[0003] For example, in fintech scenarios, existing technologies separate the modeling of generation tasks (such as personalized product descriptions and claims responses) from the modeling of representation tasks (such as risk assessment and customer segmentation), leading to a break in the financial service chain. For instance, telemarketing script generation models cannot access the latest customer risk profiles in real time, resulting in a mismatch between product recommendations and underwriting capabilities; in healthcare scenarios, existing technologies separate the modeling of generation tasks from the modeling of representation tasks, leading to a break in the healthcare service chain.

[0004] Therefore, how to implement a method for constructing large language models for multiple tasks in order to improve the efficiency of parallel task processing has become an urgent technical problem to be solved. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for constructing large language models for multiple tasks, with the main purpose of solving the problem of low efficiency in parallel processing of tasks.

[0006] Firstly, to achieve the above objectives, the present invention provides a method for constructing a large language model oriented for multiple tasks, comprising: Construct low-rank matrices for each preset task type, and configure a corresponding target key vector for each low-rank matrix; Obtain the task data corresponding to each preset task type, and extract the semantic feature vector of the task data corresponding to each preset task type; Calculate the cosine similarity between the semantic feature vector of the task data corresponding to each preset task type and each target key vector, and map the cosine similarity to the routing score used by the corresponding low-rank matrix to process the task data; At least one target low-rank matrix is ​​selected from the plurality of low-rank matrices based on the routing score; Based on the target low-rank matrix, a model representation learning process is performed on the preset large language model to obtain a multi-task large language model for parallel processing of various different task types.

[0007] Secondly, the present invention also provides a large language model construction apparatus for multi-task applications, comprising: The matrix configuration module is used to construct low-rank matrices corresponding to each preset task type, and configure a corresponding target key vector for each low-rank matrix; The feature extraction module is used to obtain the task data corresponding to each preset task type and extract the semantic feature vector of the task data corresponding to each preset task type. The routing calculation module is used to calculate the cosine similarity between the semantic feature vector of the task data corresponding to each preset task type and each target key vector, and to map the cosine similarity to the routing score used by the corresponding low-rank matrix to process the task data; A matrix selection module is used to select at least one target low-rank matrix from a plurality of low-rank matrices based on the routing score; The model learning module is used to perform model representation learning on a preset large language model based on the target low-rank matrix, so as to obtain a multi-task large language model for parallel processing of multiple different task types.

[0008] Thirdly, the present invention also provides an electronic device, the electronic device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the above-described multi-task-oriented large language model construction method.

[0009] Fourthly, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the above-described method for constructing a large language model for multitasking.

[0010] In this embodiment of the invention, by pre-constructing low-rank matrices and key vectors corresponding to task types, multi-task capabilities are decoupled into pluggable independent modules, fundamentally avoiding the catastrophic forgetting and parameter explosion problems caused by full-parameter fine-tuning. Secondly, during the inference stage, by extracting semantic feature vectors from task data and calculating cosine similarity with the key vectors, qualitative task identification is transformed into quantitative routing scores, achieving high-precision automatic task type discrimination. Then, based on the scores, the most suitable target low-rank matrix is ​​dynamically selected, enabling the model to "adapt to local conditions" and allocate the most suitable expert modules to different inputs, ensuring both computational efficiency and... Resources are used on demand, avoiding the accumulation of errors caused by rigid partitioning. Finally, by using selected low-rank matrices to perform representation learning on large language models, a single basic model can simultaneously master multiple differentiated capabilities such as mathematical reasoning, code generation, and sentiment analysis with only a minimal increase in parameters. Moreover, the low-rank matrices can be computed in parallel, significantly reducing inference latency in multi-task concurrent scenarios. For example, in fintech scenarios, this invention can model and compute generation tasks (such as personalized product descriptions and claims responses) and representation tasks (such as risk assessment and customer segmentation) in parallel to improve the efficiency of multi-task parallel processing in financial scenarios. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of an application environment for a multi-task-oriented large language model construction method according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating a method for constructing a large language model for multiple tasks according to an embodiment of the present invention. Figure 3 yes Figure 2 A flowchart illustrating a specific implementation method of step S1; Figure 4 yes Figure 2 A flowchart illustrating a specific implementation method of step S2; Figure 5 yes Figure 2 A flowchart illustrating a specific implementation method of step S3; Figure 6 yes Figure 2 A flowchart illustrating a specific implementation of step S4; Figure 7 yes Figure 2 A flowchart illustrating a specific implementation of step S5; Figure 8 yes Figure 2 A flowchart illustrating another specific implementation of step S5; Figure 9 A functional block diagram of a multi-task-oriented large language model construction device provided in an embodiment of the present invention; Figure 10 A schematic diagram of an electronic device for implementing a multi-task-oriented large language model construction method according to an embodiment of the present invention; Figure 11 This is another structural schematic diagram of an electronic device that implements a method for constructing a large language model for multi-task purposes, according to an embodiment of the present invention.

[0013] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0014] It should be noted that in the technical solutions disclosed in this invention, the acquisition of user information (personal image data (e.g., facial videos or pictures, facial feature videos or pictures, etc.) and personal privacy information (e.g., name, ID number, occupation, address, etc.)) is all completed with the user's knowledge and consent, and the acquisition of the relevant user information is legal and compliant.

[0015] To enable those skilled in the art to better understand the technical solutions of this disclosure, and to fully understand and implement the process of how this disclosure applies technical means to solve technical problems and achieve corresponding technical effects, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. The embodiments of this disclosure and the various features within them can be combined with each other without conflict, and the resulting technical solutions are all within the protection scope of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort should fall within the protection scope of this disclosure.

[0016] It should be noted that the terms "first," "second," etc., used in this disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0017] This application provides a method for constructing a large language model for multiple tasks. The execution entity of this method includes, but is not limited to, at least one electronic device that can be configured to execute the device provided in this application, such as a server or a terminal. In other words, the method for constructing a large language model for multiple tasks can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cluster of cloud servers. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0018] This invention provides a method for constructing large language models for multiple tasks, which can be applied to applications such as... Figure 1 In this application environment, the client communicates with the server via a network. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0019] Reference Figure 2 The diagram shown is a flowchart illustrating a multi-task-oriented large language model construction method according to an embodiment of the present invention. In this embodiment, the multi-task-oriented large language model construction method includes: S1. Construct low-rank matrices corresponding to each preset task type, and configure a corresponding target key vector for each low-rank matrix.

[0020] like Figure 3As shown in this embodiment of the invention, configuring a corresponding target key vector for each low-rank matrix includes: S31. Obtain the matrix dimension information and matrix statistical characteristics of each of the low-rank matrices; S32. Based on the matrix dimension information, determine the preset length of the key vector corresponding to each low-rank matrix; S33. Based on the statistical characteristics of the matrix, match the corresponding key vector generation strategy from the preset key vector generation strategy library; S34. Generate an initial key vector for each of the low-rank matrices according to the key vector generation strategy, and perform modulus normalization on the initial key vectors to obtain the target key vector.

[0021] In this embodiment of the invention, by traversing the storage structure of the matrix, the extended scale of the matrix in different dimensions is accurately extracted. At the same time, the distribution characteristics of all elements in the matrix are statistically analyzed in the whole domain, and the basic structural attributes and global numerical features of the matrix are fully captured, providing underlying data support for the subsequent generation of key vectors.

[0022] In detail, based on the structural redundancy characteristics of low-rank matrices and combined with dimension-adaptive mapping rules, the spatial extension scale of the matrix is ​​matched with the representation capacity of the key vector. While fully preserving the recognizability of the core structure of the matrix, a reasonable length range of the key vector is defined to avoid the waste of computing power caused by length redundancy, while ensuring the accuracy of subsequent semantic association.

[0023] Specifically, relying on a pre-built multi-scenario strategy indexing system, the extracted global numerical features of the matrix are used as matching anchors. The features and generation logic are precisely aligned in a strategy library covering different distribution characteristics, so that the subsequently generated key vectors can naturally adapt to the numerical distribution patterns of the corresponding low-rank matrix, thereby strengthening the feature correlation between the key vectors and the original matrix.

[0024] Furthermore, through the exclusive generation logic obtained by matching, the core features of the low-rank matrix are encoded and mapped to the initial key vector. Then, relying on the normalization constraint mechanism of the vector space, the overall extension scale of the initial key vector is uniformly calibrated to eliminate the scale difference between different vectors, so that all target key vectors are in a unified metric space, ensuring the fairness and accuracy of subsequent vector similarity calculation.

[0025] For example, in the healthcare scenario, low-rank matrices are pre-constructed for four types of medical-specific tasks: structured extraction of electronic medical records, auxiliary annotation of medical images, recommendation of clinical treatment plans, and early warning of patient health risks. For each type of low-rank matrix, a unique normalized target key vector is generated based on its dimension and statistical characteristics. The key vector can accurately match the semantic features of medical text, images, and structured health data. After connecting to the general large model, it is only necessary to quickly call the corresponding low-rank matrix to complete the lightweight adaptation of medical-specific tasks by matching the key vector with the semantic features of the task. The original general medical knowledge base of the large model does not need to be modified throughout the process.

[0026] For example, in a fintech scenario, low-rank matrices are pre-constructed to correspond to four types of financial-specific tasks: intelligent risk control and anti-fraud, extraction of research report information elements, intelligent investment research target analysis, and automatic review of compliance documents. Normalized target key vectors are generated by combining the dimensional attributes and numerical distribution characteristics of each matrix. The key vectors can accurately align with the semantic features of financial transaction data, research report texts, and compliance documents. After being connected to a general large model, the corresponding low-rank matrices can be quickly scheduled to complete the efficient adaptation of financial-specific tasks through similarity routing between task semantic features and key vectors. The original general financial cognitive capabilities of the large model are preserved throughout the process, avoiding the risk of knowledge shift caused by full-scale fine-tuning.

[0027] In this embodiment of the invention, low-rank decomposition significantly reduces the number of model parameters and computational overhead, improving multi-task inference efficiency. Simultaneously, the key vector mechanism enables the system to quickly retrieve and activate the most suitable matrix module based on input features at runtime, achieving adaptive task switching. This approach avoids the high cost of full parameter fine-tuning and alleviates knowledge conflicts and catastrophic forgetting between multiple tasks. While maintaining single-task accuracy, it greatly enhances the model's scalability and deployment flexibility, making it particularly suitable for resource-constrained edge computing and dynamic service scenarios.

[0028] S2. Obtain the task data corresponding to each preset task type, and extract the semantic feature vector of the task data corresponding to each preset task type.

[0029] like Figure 4 As shown in this embodiment of the invention, the extraction of semantic feature vectors of task data corresponding to each preset task type includes: S41. Identify the data type of the task data corresponding to each preset task type, wherein the data type includes text type, image type and structured numerical type; S42. Perform data standardization processing on the task data according to the data type to generate standardized task data; S43. If the data type of the standardized task data is text, then the standardized task data is encoded into text using a preset semantic encoding model to obtain a text semantic feature vector. S44. If the data type of the standardized task data is an image type, then the standardized task data is visually encoded using a preset visual encoding model to obtain an image semantic feature vector. S45. If the data type of the standardized task data is a structured numerical type, then the standardized task data is normalized and its features are concatenated to obtain a structured semantic feature vector. S46. The text semantic feature vector, the image semantic feature vector, and the structured semantic feature vector are summarized into the semantic feature vector of the task data.

[0030] In this embodiment of the invention, by performing full-domain verification on the underlying storage format, semantic carrying form and structural attributes of task data, the modal category to which the data belongs is accurately determined, and the initial diversion of data with different attributes is completed, providing a preliminary judgment basis for subsequent differentiated processing.

[0031] In detail, based on the inherent characteristics of different modal data, corresponding format regularization and attribute alignment strategies are matched to uniformly correct the data storage specifications, expression forms and structural boundaries, eliminate interference problems such as format deviations and attribute misalignments in the original data, and obtain standardized task data that is suitable for subsequent coding processes.

[0032] The process involves mapping standardized text data to a high-dimensional semantic space, capturing the contextual relationships and deep semantic logic within the text through a multi-layer attention mechanism, completing the transformation from natural language to structured semantic representation, and outputting a text semantic feature vector with complete semantic information.

[0033] The process involves calling a pre-defined visual encoding model and capturing edge textures, local semantics, and global scene information in an image layer by layer through a multi-level feature extraction network. This transforms the pixel-level visual signals into a unified high-dimensional semantic representation, resulting in an image semantic feature vector that can fully carry the core semantics of the image.

[0034] Among them, the dimensional differences between different numerical dimensions are eliminated by the global distribution calibration mechanism, so that all numerical features are under a unified representation scale. Then, the scattered dimensional features are sequentially linked according to the preset feature arrangement logic and integrated into a structured semantic feature vector that can fully cover the core information of structured data.

[0035] Furthermore, relying on the multimodal feature fusion mechanism, the semantic feature vectors generated by the three different modalities are aggregated in an orderly manner in a unified feature space. While retaining the unique semantic information of each modality, cross-modal semantic complementarity is achieved, and finally a unified semantic feature vector that can fully cover the core semantics of the entire task data is obtained.

[0036] For example, in a healthcare scenario, data from four different types of medical tasks are collected in batches: outpatient electronic medical records, lung CT images, structured forms for chronic disease follow-up, and patient complaints transcribed from speech. First, the format of the multimodal data is standardized and cleaned. Then, a unified semantic feature vector covering the core semantics of the full-modal medical data is extracted through a text semantic encoding model adapted to medical terminology, a medical image visual encoding model, and a structured numerical normalization splicing process. This vector fully carries the exclusive semantic information of different types of medical tasks.

[0037] For example, in a fintech scenario, data from four different types of financial tasks are collected in batches: credit application risk control texts, publicly available research reports from listed companies, structured transaction data, and scanned images of compliant contracts. First, the format of the cross-modal financial data is standardized and noise is removed. Then, a unified semantic feature vector covering the core semantics of the full-modal financial data is extracted through a text semantic encoding model adapted to financial terminology, a visual encoding model for financial documents, and a structured numerical normalization splicing process. This vector fully carries the exclusive semantic information of different types of financial tasks.

[0038] In this embodiment of the invention, heterogeneous task data (such as text, images, and code) are transformed into computable vector forms through a unified semantic representation space, providing standardized input for subsequent task recognition and model adaptation. This enables the system to quantify the similarity and differences between different tasks, thereby guiding the accurate selection and dynamic routing of low-rank matrices.

[0039] S3. Calculate the cosine similarity between the semantic feature vector of the task data corresponding to each preset task type and each target key vector, and map the cosine similarity to the routing score used by the corresponding low-rank matrix to process the task data.

[0040] like Figure 5 As shown in this embodiment of the invention, mapping the cosine similarity to the routing score used to process the task data corresponding to the low-rank matrix includes: S51. Normalize the cosine similarity of each matrix using a preset activation function to obtain the first routing score of each low-rank matrix. S52. Obtain the task complexity coefficient of the task data, and correct the first routing score according to the task complexity coefficient to obtain the corrected second routing score of each of the low-rank matrices. S53. Compare the second routing score with the preset minimum routing threshold; S54. Use the second routing score, which is greater than or equal to the minimum routing threshold, as the routing score used to process the task data corresponding to the low-rank matrix. S55. Set the second routing score, which is lower than the minimum routing threshold, to zero and renormalize it. Use the renormalized second routing score as the routing score used to process the task data corresponding to the low-rank matrix.

[0041] In this embodiment of the invention, relying on the probability mapping mechanism of the Softmax function, the originally unbounded cosine similarity values ​​are uniformly converted into normalized representations within a standard range. While preserving the original relative similarity, the differences in matching degree between different low-rank matrices and task data are amplified to obtain a first routing score with basic probability distribution attributes.

[0042] Specifically, by analyzing the multi-dimensional attributes of the task data, a complexity coefficient that can accurately represent the difficulty of task processing is extracted. This coefficient is then used as a control factor to apply to the first routing score. The weight ratio of different low-rank matrices is dynamically adjusted according to the actual computing power requirements of the task, so that the distribution characteristics of the routing score are accurately matched with the complexity requirements of the task, and the corrected second routing score is obtained.

[0043] Furthermore, based on the preset route admission judgment rules, all the corrected second route scores are compared with the minimum route threshold one by one across the entire domain to complete the initial screening of low matching degree route branches, and mark the low-rank matrix that has not reached the basic service admission threshold, providing a clear judgment basis for subsequent route branch pruning operations.

[0044] Among them, the second routing score that meets the threshold requirement after comparison is directly retained as the final routing score of the corresponding low-rank matrix. The matching degree of the low-rank matrix corresponding to this score with the current task reaches the preset standard, and it can directly participate in the subsequent task allocation and scheduling, ensuring that the routing priority of the high-adaptability matrix is ​​not disturbed by additional adjustments.

[0045] Specifically, the second route score that has not reached the minimum route threshold is cleared to zero, completely eliminating the participation qualification of low-rank matrices with excessively low matching scores. Then, a second normalization calibration is performed on the remaining valid scores to bring the final route scores back to the legal probability distribution range, ensuring the stability and rationality of subsequent task route scheduling.

[0046] In this embodiment of the invention, cosine similarity is used to accurately measure the matching degree between the input task and the predefined task type in geometric space, transforming qualitative task identification into a quantitatively calculable similarity score. Then, the similarity score is converted into a normalized routing score using mapping functions such as Softmax, achieving soft, differentiable allocation of tasks to the matrix module. This routing mechanism ensures both the hard accuracy of task allocation and the ability to backpropagate gradients, enabling the entire system to support end-to-end joint optimization, adaptively handle mixed tasks, and significantly improve robustness and generalization performance in multi-task scenarios.

[0047] S4. Select at least one target low-rank matrix from the plurality of low-rank matrices according to the routing score.

[0048] like Figure 6 As shown in this embodiment of the invention, selecting at least one target low-rank matrix from the plurality of low-rank matrices based on the routing score includes: S61. Sort all the route scores in descending order to obtain a route score sorting list; S62. Based on the preset target selection quantity, select the top K low-rank matrices from the routing score sorting list as preliminary candidate matrices, where K is a positive integer and is less than or equal to the total number of low-rank matrices. S63. Calculate the cumulative percentage of routing scores for each of the preliminary candidate matrices; S64. If the cumulative percentage of the routing score is less than a preset cumulative percentage threshold, then the candidate matrix is ​​added in descending order of the routing score until the cumulative percentage of the routing score is greater than or equal to the cumulative percentage threshold, thus obtaining an extended candidate matrix set. S65. Obtain the matrix load state of each low-rank matrix in the extended candidate matrix set, and remove low-rank matrices whose matrix load state is greater than a preset load threshold to obtain an available candidate matrix set. S66. Select at least one target low-rank matrix from the available candidate matrix set according to the routing score from high to low.

[0049] In this embodiment of the invention, relying on the global sorting mechanism, the routing scores corresponding to all low-rank matrices are traversed and prioritized, and the sequence is properly normalized in strict accordance with the priority from high to low scores, generating a complete routing score sorting list that clearly presents the adaptation priority level of different low-rank matrices and the current task.

[0050] Specifically, based on the pre-set target selection number rules, a specified number of high-priority low-rank matrices are extracted from the head of the sorted routing score list as preliminary candidate matrices. Under the premise of ensuring core adaptability, the first round of candidate matrix screening is completed, narrowing down the matrix range for subsequent processing.

[0051] Furthermore, relying on the proportion statistics mechanism, the routing scores of all preliminary candidate matrices are accumulated layer by layer. The proportion of the cumulative score of each newly added candidate matrix in the total score is calculated, so as to accurately grasp the overall weight coverage of the current candidate set and provide a basis for judgment for subsequent candidate matrix expansion operations.

[0052] Using a preset cumulative percentage threshold as the criterion, when the cumulative weight coverage of the existing preliminary candidate matrix does not meet the standard, it is sequentially included in the low-rank matrix of the subsequent sorting position according to the routing score from high to low, until the cumulative weight coverage meets the threshold requirement, forming an extended candidate matrix set with the coverage standard.

[0053] In detail, the current running load status of all low-rank matrices in the extended candidate matrix set is collected in real time. The load index of each matrix is ​​compared with the preset load threshold one by one. Overloaded matrices whose load exceeds the safe threshold are eliminated, and the available candidate matrix set that is currently in a normal schedulable state is selected.

[0054] Within the set of available candidate matrices that have been screened, the priority rule of routing scores from high to low is strictly followed, and at least one low-rank matrix with the best fit is selected as the final target low-rank matrix to complete the final node output of the entire routing scheduling process.

[0055] In this embodiment of the invention, by sorting by routing scores or filtering by threshold, computing resources are dynamically allocated on demand—high-score matrices are activated first to participate in inference, while low-score matrices are skipped or downweighted. This significantly reduces floating-point operations and memory usage while ensuring task accuracy, enabling the system to have task adaptability and flexibly adjust the set of activation matrices according to different inputs. This avoids the performance bottleneck of fixed architectures in multi-task scenarios and provides an efficient and elastic computing solution for large-scale concurrent services.

[0056] S5. Based on the target low-rank matrix, perform model representation learning on the preset large language model to obtain a multi-task large language model for parallel processing of various different task types.

[0057] like Figure 7 As shown in this embodiment of the invention, the step of performing model representation learning on a preset large language model based on the target low-rank matrix to obtain a multi-task large language model for parallel processing of multiple different task types includes: S71. The target low-rank matrix is ​​embedded into multiple network layers of the preset large language model as low-rank adaptation parameters for each network layer. S72. Obtain multi-task training datasets corresponding to various different task types, and divide the multi-task training datasets into multiple task-specific subsets according to the task type; S73. Divide each task-specific subset into a training subset and a validation subset; S74. Perform forward propagation calculation on the large language model based on the training subset and the low-rank adaptation parameters to obtain the initial loss value of the large language model; S75. The large language model is updated by backpropagation based on the initial loss value to obtain an intermediate large language model; S76. Perform forward propagation calculation on the intermediate large language model based on the validation subset to obtain the target loss value of the intermediate large language model; S77. If the target loss value is less than the preset loss threshold, the intermediate large language model is used as the current large language model to continue to perform backpropagation update and forward propagation calculation until the target loss value is greater than or equal to the loss threshold. Then, the intermediate large language model corresponding to the target loss value is used as the multi-task large language model.

[0058] In this embodiment of the invention, relying on the plug-in embedding mechanism of low-rank adaptation, the selected target low-rank matrix is ​​precisely injected into multiple core network layers of the model on the basis of freezing the original pre-trained weights of the large language model. As an adaptation parameter module that can be trained independently, it does not destroy the original general knowledge system of the model, and can reserve lightweight adjustment space for subsequent multi-task adaptation.

[0059] In detail, relying on the semantic classification mechanism of multimodal tasks, semantic label recognition is performed on the collected full-domain training data covering different task types. The data is accurately divided according to the core attributes and processing logic of the task, and different types of task data are grouped into mutually independent task-specific subsets to avoid semantic interference between cross-task data and ensure the purity of subsequent single-task adaptation.

[0060] Among them, relying on the stratified sampling dataset splitting mechanism, while preserving the data distribution characteristics within each task-specific subset, the subset is split into two non-overlapping parts according to a preset reasonable ratio. The training subset is used for subsequent model adaptation learning, and the verification subset is used to independently verify the model's generalization ability on the task, thus avoiding the risk of overfitting during the training process.

[0061] Specifically, the input data of the training subset is fed into the model to complete the layer-by-layer feature transformation and semantic mapping. Task-specific feature fine-tuning is completed only through the newly added low-rank adaptation parameters. Finally, the model outputs the prediction result of the current task. The initial loss value is generated based on the deviation between the prediction result and the true label, which provides optimization direction guidance for subsequent parameter updates.

[0062] Furthermore, the optimization signal of the initial loss value is backpropagated layer by layer, and only the low-rank adaptation parameters of the embedding are updated with gradients. The original pre-trained weights of the large language model are kept constant throughout the process. While significantly reducing the computational cost, the parameter optimization of this iteration is completed, and an intermediate large language model adapted to the features of the current batch of tasks is output.

[0063] Furthermore, the validation subset that was not used in training is input into the current intermediate large language model, and forward feature calculation is completed in the inference mode without gradient update. The target loss value is generated based on the prediction bias of the model on independent validation data, which accurately represents the generalization performance of the current model on the corresponding task and provides an objective basis for determining the training termination condition.

[0064] In detail, a preset loss threshold is used as the criterion for training convergence. When the target loss value does not meet the threshold requirement, the iterative process of forward propagation calculation and backpropagation update is executed repeatedly to continuously optimize the adaptation effect of low-rank adaptation parameters until the target loss value meets the convergence criterion. Finally, a multi-task large language model that can be adapted in parallel to various different task types is output.

[0065] like Figure 8 As shown in this embodiment of the invention, the step of performing forward propagation calculation on the large language model based on the training subset and the low-rank adaptation parameters to obtain the initial loss value of the large language model includes: S81. Obtain several training batch samples from the training subset, wherein the training batch samples include an input text sequence and a corresponding real label sequence, and convert the input text sequence into an input tensor; S82. Extract the original weight matrix and low-rank adaptation parameter matrix contained in the first network layer of the large language model; S83. Perform matrix multiplication operations on the input tensor in parallel with the original weight matrix and the low-rank adaptation parameter matrix to obtain the original output features and the low-rank increment features. Add the original output features and the low-rank increment features to obtain the layer output features of the first network layer. S84. The output features of the layer are passed to the next network layer, and the matrix multiplication operation is repeated in each network layer containing the low-rank adaptation parameters until the input tensor passes through all network layers of the large language model in sequence to obtain the predicted output probability distribution corresponding to the input text sequence. S85. Using a preset loss function, calculate the initial loss value of the large language model on the training subset based on the predicted output probability distribution and the real label sequence.

[0066] In this embodiment of the invention, a set of training samples is extracted from the training subset, and the input text sequence and corresponding real label sequence within the sample are completely preserved. Then, through the vectorization mapping technology of text semantics, the discrete natural language text is converted into a structured input tensor that can be recognized by the deep learning framework, thus completing the conversion of unstructured text data into a representation form that the model can process.

[0067] In detail, the first network layer of the large language model with embedded low-rank adaptation module is located. The original weight matrix solidified in the pre-training stage and the subsequently injected trainable low-rank adaptation parameter matrix are extracted from the parameter storage space of this layer to prepare all the core parameter carriers for subsequent parallel feature calculation.

[0068] The input tensor is fed into two independent computation branches simultaneously to perform feature transformation operations with the original weight matrix and the low-rank adaptation parameter matrix, respectively, to obtain the basic features of the original path output and the incremental adaptation features of the low-rank path output. The two types of features are then semantically fused to generate the final layer output features of the first network layer.

[0069] Furthermore, the fused features output from the first network layer are passed to the next network layer. The dual-path parallel feature calculation process is repeated in all network layers that embed low-rank adaptation parameters, allowing the input tensor to complete semantic transformation layer by layer, and finally outputting the predicted output probability distribution of the corresponding input text sequence.

[0070] Furthermore, the predicted probability distribution output by the model is compared with the true label sequence of the sample to perform a global semantic deviation comparison, accurately quantifying the prediction error of the current model on this batch of training samples, and finally obtaining the initial loss value of the large language model on the training subset, providing a clear optimization anchor point for subsequent backpropagation parameter optimization.

[0071] For example, in a healthcare scenario, the four types of medical-specific target low-rank matrices selected are embedded into the multi-layer Transformer network layer of a general medical model as adaptation parameters. Molecular set training and iterative verification are completed using annotated multi-task medical datasets. Representation learning is completed while freezing the original pre-trained medical knowledge of the large model throughout the process. Finally, a multi-task medical model capable of parallel processing of multiple tasks such as electronic medical record extraction, image-assisted annotation, treatment plan recommendation, and health risk warning is obtained, balancing the retention of general medical knowledge and the efficiency of multi-task adaptation.

[0072] For example, in a fintech scenario, the four types of low-rank matrices of financial-specific targets selected are embedded into the multi-layer Transformer network layer of a general financial model as adaptation parameters. The molecular set training and iterative verification are completed using an annotated multi-task financial dataset. Representation learning is completed under the premise of freezing the original pre-trained financial knowledge of the large model throughout the process. Finally, a multi-task financial model that can process multiple tasks in parallel, such as risk control and anti-fraud, research report element extraction, investment research target analysis, and compliance text review, is obtained, which takes into account both general financial knowledge retention and multi-task processing performance.

[0073] In this embodiment of the invention, the model weights are efficiently fine-tuned in a structured manner using low-rank matrices. Learnable low-rank increments are added only to the original pre-trained weights, which not only preserves the powerful general knowledge representation ability of the basic model, but also learns the feature mappings specific to each task in parallel through multiple sets of low-rank matrices. This allows a single model to activate the corresponding matrix as needed during inference, enabling parallel or interleaved processing of different task types.

[0074] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0075] like Figure 9 The diagram shown is a functional block diagram of a large language model construction device for multi-task applications provided in an embodiment of the present invention.

[0076] This disclosure provides a multi-task-oriented large language model construction apparatus, which corresponds one-to-one with the multi-task-oriented large language model construction method described in the above embodiments. For example... Figure 3 As shown, this multi-task-oriented large language model construction device 100 can be installed in an electronic device. According to its functions, the multi-task-oriented large language model construction device 100 includes a matrix configuration module 101, a feature extraction module 102, a route calculation module 103, a matrix selection module 104, and a model learning module 105. Detailed descriptions of each functional module are as follows: The matrix configuration module 101 is used to construct low-rank matrices corresponding to each preset task type, and configure a corresponding target key vector for each low-rank matrix; The feature extraction module 102 is used to obtain the task data corresponding to each preset task type and extract the semantic feature vector of the task data corresponding to each preset task type. The routing calculation module 103 is used to calculate the cosine similarity between the semantic feature vector of the task data corresponding to each preset task type and each target key vector, and to map the cosine similarity to the routing score used by the corresponding low-rank matrix to process the task data; The matrix selection module 104 is used to select at least one target low-rank matrix from a plurality of low-rank matrices based on the routing score; The model learning module 105 is used to perform model representation learning on the preset large language model based on the target low-rank matrix, so as to obtain a multi-task large language model for parallel processing of multiple different task types.

[0077] In one embodiment, when the matrix configuration module 101 configures a corresponding target key vector for each of the low-rank matrices, it is used to: Obtain the matrix dimension information and matrix statistical characteristics of each of the low-rank matrices; Based on the matrix dimension information, determine the preset length of the key vector corresponding to each low-rank matrix; Based on the statistical characteristics of the matrix, a corresponding key vector generation strategy is matched from a preset key vector generation strategy library; An initial key vector is generated for each low-rank matrix according to the key vector generation strategy, and the initial key vector is normalized to obtain the target key vector.

[0078] In one embodiment, when the feature extraction module 102 extracts the semantic feature vector of the task data corresponding to each preset task type, it is used to: Identify the data type of the task data corresponding to each preset task type, wherein the data type includes text type, image type and structured numerical type; The task data is standardized according to the data type to generate standardized task data. If the data type of the standardized task data is text, then the standardized task data is encoded into text using a preset semantic encoding model to obtain a text semantic feature vector. If the data type of the standardized task data is an image, then the standardized task data is visually encoded using a preset visual encoding model to obtain an image semantic feature vector. If the data type of the standardized task data is a structured numerical type, then the standardized task data is normalized and its features are concatenated to obtain a structured semantic feature vector. The text semantic feature vector, the image semantic feature vector, and the structured semantic feature vector are summarized into the semantic feature vector of the task data.

[0079] In one embodiment, when the routing calculation module 103 performs the routing score used to process the task data by mapping the cosine similarity to the corresponding low-rank matrix, it is used to: The cosine similarity scores are normalized using a preset activation function to obtain the first routing score of each low-rank matrix. Obtain the task complexity coefficient of the task data, and correct the first route score according to the task complexity coefficient to obtain the corrected second route score of each of the low-rank matrices; The second route score is compared with a preset minimum route threshold; The second routing score, which is greater than or equal to the minimum routing threshold, is used as the routing score for processing the task data corresponding to the low-rank matrix. The second routing score, which is below the minimum routing threshold, is set to zero and renormalized. The renormalized second routing score is then used as the routing score for processing the task data corresponding to the low-rank matrix.

[0080] In one embodiment, when the matrix selection module 104 performs the operation of selecting at least one target low-rank matrix from the plurality of low-rank matrices based on the routing score, it is configured to: Sort all the route scores in descending order to obtain a sorted list of route scores; Based on the preset target selection quantity, the top K low-rank matrices from the routing score sorting list are selected as preliminary candidate matrices, where K is a positive integer and is less than or equal to the total number of low-rank matrices; Calculate the cumulative percentage of routing scores for each of the preliminary candidate matrices; If the cumulative percentage of the routing score is less than a preset cumulative percentage threshold, then the candidate matrix is ​​added in descending order of the routing score until the cumulative percentage of the routing score is greater than or equal to the cumulative percentage threshold, thus obtaining an extended candidate matrix set. Obtain the matrix load state of each low-rank matrix in the extended candidate matrix set, and remove low-rank matrices whose matrix load state is greater than a preset load threshold to obtain an available candidate matrix set. At least one target low-rank matrix is ​​selected from the set of available candidate matrices according to the routing score from high to low.

[0081] In one embodiment, when the model learning module 105 performs model representation learning on a preset large language model based on the target low-rank matrix to obtain a multi-task large language model for parallel processing of multiple different task types, it is used to: The target low-rank matrix is ​​embedded into multiple network layers of the preset large language model, respectively, as the low-rank adaptation parameter of each network layer; Obtain multi-task training datasets corresponding to various different task types, and divide the multi-task training datasets into multiple task-specific subsets according to the task type; Each task-specific subset is divided into a training subset and a validation subset; The large language model is forward propagated based on the training subset and the low-rank adaptation parameters to obtain the initial loss value of the large language model. The large language model is updated by backpropagation based on the initial loss value to obtain an intermediate large language model. Based on the validation subset, the intermediate large language model is forward propagated to obtain the target loss value of the intermediate large language model; If the target loss value is less than the preset loss threshold, the intermediate large language model is used as the current large language model to continue backpropagation update and forward propagation calculation until the target loss value is greater than or equal to the loss threshold. Then, the intermediate large language model corresponding to the target loss value is used as the multi-task large language model.

[0082] In one embodiment, when the model learning module 105 performs forward propagation calculation on the large language model based on the training subset and the low-rank adaptation parameters to obtain the initial loss value of the large language model, it is used to: Several training batch samples are obtained from the training subset, each training batch sample containing an input text sequence and a corresponding real label sequence, and the input text sequence is converted into an input tensor; Extract the original weight matrix and low-rank adaptation parameter matrix contained in the first network layer of the large language model; The input tensor is multiplied in parallel with the original weight matrix and the low-rank adaptation parameter matrix to obtain the original output features and the low-rank increment features. The original output features and the low-rank increment features are then added together to obtain the layer output features of the first network layer. The output features of the layer are passed to the next network layer, and the matrix multiplication operation is repeated in each network layer containing the low-rank adaptation parameters until the input tensor passes through all network layers of the large language model in sequence to obtain the predicted output probability distribution corresponding to the input text sequence. Using a preset loss function, the initial loss value of the large language model on the training subset is calculated based on the predicted output probability distribution and the real label sequence.

[0083] In this invention, the specific limitations of the multi-task-oriented large language model construction device can be found in the above-described limitations of the multi-task-oriented large language model construction method, and will not be repeated here. Each module in the aforementioned multi-task-oriented large language model construction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0084] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the server-side functions or steps of a multi-tasking large language model construction method.

[0085] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements client-side functions or steps of a multi-tasking, large language model construction method.

[0086] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Construct low-rank matrices for each preset task type, and configure a corresponding target key vector for each low-rank matrix; Obtain the task data corresponding to each preset task type, and extract the semantic feature vector of the task data corresponding to each preset task type; Calculate the cosine similarity between the semantic feature vector of the task data corresponding to each preset task type and each target key vector, and map the cosine similarity to the routing score used by the corresponding low-rank matrix to process the task data; At least one target low-rank matrix is ​​selected from the plurality of low-rank matrices based on the routing score; Based on the target low-rank matrix, a model representation learning process is performed on the preset large language model to obtain a multi-task large language model for parallel processing of various different task types.

[0087] In the several embodiments provided by this invention, it should be understood that the disclosed devices and apparatuses can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0088] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0089] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0090] In some embodiments of this example, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in the above embodiments.

[0091] The readable storage medium of the present invention stores a computer program, which, when executed by a processor of an electronic device, can perform the following: Construct low-rank matrices for each preset task type, and configure a corresponding target key vector for each low-rank matrix; Obtain the task data corresponding to each preset task type, and extract the semantic feature vector of the task data corresponding to each preset task type; Calculate the cosine similarity between the semantic feature vector of the task data corresponding to each preset task type and each target key vector, and map the cosine similarity to the routing score used by the corresponding low-rank matrix to process the task data; At least one target low-rank matrix is ​​selected from the plurality of low-rank matrices based on the routing score; Based on the target low-rank matrix, a model representation learning process is performed on the preset large language model to obtain a multi-task large language model for parallel processing of various different task types.

[0092] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0093] Computer-readable storage media may also store at least one computer-executable program / instruction, such as computer-readable instructions. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Computer-readable storage media may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0094] In addition, the computer device may include (but is not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (e.g., keyboard, mouse, speakers, etc.).

[0095] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product / computer program product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0096] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Furthermore, any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0097] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0098] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0099] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0100] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

Claims

1. A method for constructing a large language model for multiple tasks, characterized in that, The method includes: Construct low-rank matrices for each preset task type, and configure a corresponding target key vector for each low-rank matrix; Obtain the task data corresponding to each preset task type, and extract the semantic feature vector of the task data corresponding to each preset task type; Calculate the cosine similarity between the semantic feature vector of the task data corresponding to each preset task type and each target key vector, and map the cosine similarity to the routing score used by the corresponding low-rank matrix to process the task data; At least one target low-rank matrix is ​​selected from the plurality of low-rank matrices based on the routing score; Based on the target low-rank matrix, a model representation learning process is performed on the preset large language model to obtain a multi-task large language model for parallel processing of various different task types.

2. The method for constructing a large language model for multiple tasks as described in claim 1, characterized in that, The step of configuring a corresponding target key vector for each of the low-rank matrices includes: Obtain the matrix dimension information and matrix statistical characteristics of each of the low-rank matrices; Based on the matrix dimension information, determine the preset length of the key vector corresponding to each low-rank matrix; Based on the statistical characteristics of the matrix, a corresponding key vector generation strategy is matched from a preset key vector generation strategy library; An initial key vector is generated for each low-rank matrix according to the key vector generation strategy, and the initial key vector is normalized to obtain the target key vector.

3. The method for constructing a large language model for multiple tasks as described in claim 1, characterized in that, The extraction of semantic feature vectors for task data corresponding to each preset task type includes: Identify the data type of the task data corresponding to each preset task type, wherein the data type includes text type, image type and structured numerical type; The task data is standardized according to the data type to generate standardized task data. If the data type of the standardized task data is text, then the standardized task data is encoded into text using a preset semantic encoding model to obtain a text semantic feature vector. If the data type of the standardized task data is an image, then the standardized task data is visually encoded using a preset visual encoding model to obtain an image semantic feature vector. If the data type of the standardized task data is a structured numerical type, then the standardized task data is normalized and its features are concatenated to obtain a structured semantic feature vector. The text semantic feature vector, the image semantic feature vector, and the structured semantic feature vector are summarized into the semantic feature vector of the task data.

4. The method for constructing a large language model for multiple tasks as described in claim 1, characterized in that, The step of mapping the cosine similarity to the routing score used to process the task data corresponding to the low-rank matrix includes: The cosine similarity scores are normalized using a preset activation function to obtain the first routing score of each low-rank matrix. Obtain the task complexity coefficient of the task data, and correct the first route score according to the task complexity coefficient to obtain the corrected second route score of each of the low-rank matrices; The second route score is compared with a preset minimum route threshold; The second routing score, which is greater than or equal to the minimum routing threshold, is used as the routing score for processing the task data corresponding to the low-rank matrix. The second routing score, which is below the minimum routing threshold, is set to zero and renormalized. The renormalized second routing score is then used as the routing score for processing the task data corresponding to the low-rank matrix.

5. The method for constructing a large language model for multiple tasks as described in claim 1, characterized in that, The step of selecting at least one target low-rank matrix from the plurality of low-rank matrices based on the routing score includes: Sort all the route scores in descending order to obtain a sorted list of route scores; Based on the preset target selection quantity, the top K low-rank matrices from the routing score sorting list are selected as preliminary candidate matrices, where K is a positive integer and is less than or equal to the total number of low-rank matrices; Calculate the cumulative percentage of routing scores for each of the preliminary candidate matrices; If the cumulative percentage of the routing score is less than a preset cumulative percentage threshold, then the candidate matrix is ​​added in descending order of the routing score until the cumulative percentage of the routing score is greater than or equal to the cumulative percentage threshold, thus obtaining an extended candidate matrix set. Obtain the matrix load state of each low-rank matrix in the extended candidate matrix set, and remove low-rank matrices whose matrix load state is greater than a preset load threshold to obtain an available candidate matrix set. At least one target low-rank matrix is ​​selected from the set of available candidate matrices according to the routing score from high to low.

6. The method for constructing a large language model for multiple tasks as described in claim 1, characterized in that, The step of learning model representations for a pre-defined large language model based on the target low-rank matrix to obtain a multi-task large language model for parallel processing of various task types includes: The target low-rank matrix is ​​embedded into multiple network layers of the preset large language model, respectively, as the low-rank adaptation parameter of each network layer; Obtain multi-task training datasets corresponding to various different task types, and divide the multi-task training datasets into multiple task-specific subsets according to the task type; Each task-specific subset is divided into a training subset and a validation subset; The large language model is forward propagated based on the training subset and the low-rank adaptation parameters to obtain the initial loss value of the large language model. The large language model is updated by backpropagation based on the initial loss value to obtain an intermediate large language model. Based on the validation subset, the intermediate large language model is forward propagated to obtain the target loss value of the intermediate large language model; If the target loss value is less than the preset loss threshold, the intermediate large language model is used as the current large language model to continue backpropagation update and forward propagation calculation until the target loss value is greater than or equal to the loss threshold. Then, the intermediate large language model corresponding to the target loss value is used as the multi-task large language model.

7. The method for constructing a large language model for multiple tasks as described in claim 6, characterized in that, The step of performing forward propagation calculation on the large language model based on the training subset and the low-rank adaptation parameters to obtain the initial loss value of the large language model includes: Several training batch samples are obtained from the training subset, each training batch sample containing an input text sequence and a corresponding real label sequence, and the input text sequence is converted into an input tensor; Extract the original weight matrix and low-rank adaptation parameter matrix contained in the first network layer of the large language model; The input tensor is multiplied in parallel with the original weight matrix and the low-rank adaptation parameter matrix to obtain the original output features and the low-rank increment features. The original output features and the low-rank increment features are then added together to obtain the layer output features of the first network layer. The output features of the layer are passed to the next network layer, and the matrix multiplication operation is repeated in each network layer containing the low-rank adaptation parameters until the input tensor passes through all network layers of the large language model in sequence to obtain the predicted output probability distribution corresponding to the input text sequence. Using a preset loss function, the initial loss value of the large language model on the training subset is calculated based on the predicted output probability distribution and the real label sequence.

8. A large language model construction device for multi-task applications, characterized in that, The device includes: The matrix configuration module is used to construct low-rank matrices corresponding to each preset task type, and configure a corresponding target key vector for each low-rank matrix; The feature extraction module is used to obtain the task data corresponding to each preset task type and extract the semantic feature vector of the task data corresponding to each preset task type. The routing calculation module is used to calculate the cosine similarity between the semantic feature vector of the task data corresponding to each preset task type and each target key vector, and to map the cosine similarity to the routing score used by the corresponding low-rank matrix to process the task data; A matrix selection module is used to select at least one target low-rank matrix from a plurality of low-rank matrices based on the routing score; The model learning module is used to perform model representation learning on a preset large language model based on the target low-rank matrix, so as to obtain a multi-task large language model for parallel processing of multiple different task types.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the multi-task-oriented large language model construction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the large language model construction method for multi-task applications as described in any one of claims 1 to 7.

Images