A large model multi-domain joint fine-tuning method, device, equipment and storage medium

Abstract

The embodiment of the application discloses a large model multi-field joint fine-tuning method, device, equipment and storage medium, which comprises the following steps: splitting training data to obtain multi-field sub-data sets, fine-tuning a first base model, generating Token first importance score according to an importance perception module, performing weighted operation on self-attention and a loss function, and generating each field expert model. A large-capacity second base model is used as a student model, and a multi-dimensional importance perception module and a field gate module are configured in the Transformer layer; the expert model is used as a teacher to carry out joint distillation, Token field importance score and field gate probability are generated, and unified fusion importance score is obtained by fusion. Based on the fusion score, the attention of the student model is weighted, the loss is calculated, the distillation constraint is applied to the corresponding expert model of the matching sample, and the target large model is obtained after training. The scheme can improve the precision of the large model in multiple vertical scenes.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and large model fine-tuning technology, and in particular to a method, apparatus, device and storage medium for multi-domain joint fine-tuning of large models. Background Technology

[0002] The Qwen series, DeepSeek series, and other general dense pre-trained large models have excellent general text semantic understanding and generation capabilities, but they have inherent defects when adapting to vertical professional business scenarios. The models have weak ability to recognize industry-specific professional terms, have difficulty grasping the business logic of specific sub-fields, and are prone to confusion between different domain knowledge when handling cross-domain mixed questions. The output content is too general and cannot meet the hard requirements of content accuracy in professional scenarios.

[0003] Current mainstream supervised fine-tuning (SFT) methods rely solely on question-and-answer samples for training, and their optimization objectives are limited to the basic output loss at the token level. They do not design corresponding constraint mechanisms for domain semantic feature alignment, text key information weight enhancement, and multi-domain knowledge boundary differentiation. In multi-domain mixed business scenarios, models fine-tuned in this way will frequently exhibit defects such as ambiguous domain attribution judgment, crosstalk between professional knowledge from different industries, and non-standard output of professional sentence structures and normative expressions.

[0004] While optimization techniques such as ultra-long contextual attention, sparse multi-expert architecture, knowledge feature distillation, and token importance weighting have been widely applied to large model pre-training and fine-tuning processes, existing technologies are unable to solve the core pain points of deploying dense base models to multiple vertical business domains. Dense base models lack lightweight and low-cost multi-domain adaptive adaptation solutions, cannot achieve automatic domain recognition for text input, and cannot complete the dynamic and accurate transfer of professional knowledge from different domains.

[0005] Existing large model fine-tuning techniques have four core shortcomings: First, conventional fine-tuning methods mix samples from all business domains and train them uniformly without differentiating the model based on the data characteristics of each business. The professional knowledge of different industries interferes with and confuses each other during the training process, and the model cannot uniformly adapt to the output format and professional expression standards corresponding to various businesses.

[0006] Second, the traditional fine-tuning process uses equal weights for all text tokens in loss calculation, which fails to amplify the learning weights of core key content such as numerical parameters, industry terminology, and legal provisions. The trained model is prone to outputting incorrect values ​​and citing incorrect legal provisions, and may also generate unfounded false content, i.e., generating illusions.

[0007] Third, the dense Transformer base model, represented by Qwen3-8B, adopts a fully parameter-shared activation architecture, and the model lacks an explicit domain recognition module and a dynamic knowledge routing mechanism. Although this architecture has outstanding performance in general language understanding, when conducting multi-domain joint fine-tuning, the knowledge of different industries will couple and interfere with each other within the shared parameter space, and the model cannot dynamically retrieve the professional knowledge of the corresponding domain to complete the inference based on the current input task.

[0008] Fourth, traditional fine-tuning only uses a single cross-entropy loss function or a composite loss function with fixed weights, which cannot simultaneously take into account the three types of requirements: domain classification and recognition, consistency of professional knowledge output, and collaborative optimization across multiple business scenarios. During multi-domain joint training, problems such as mutual coverage of domain knowledge, drift of model domain attributes, and reduced stability of text generation effects may occur. Summary of the Invention

[0009] This invention provides a method, apparatus, device, and storage medium for multi-domain joint fine-tuning of large models. Through domain expert fine-tuning, token bidirectional importance weighting, domain-gated soft routing, and multi-loss collaborative distillation architecture, dense large models can automatically distinguish business domains, strengthen the learning of professional key content, eliminate cross-domain knowledge interference, and steadily improve the generation accuracy of multiple vertical scenarios.

[0010] According to one aspect of the present invention, a multi-domain joint fine-tuning method for a large model is provided, comprising: Acquire training data and divide the training data into at least two domain subsets according to a preset domain division standard; At least two first base models are obtained. For each first base model, the first base model is fine-tuned using the corresponding domain subset dataset. During the fine-tuning training process, the first importance score of each token in the current processing sequence is generated through the importance awareness module. The self-attention calculation and loss calculation of the first base model are weighted based on the first importance score to obtain the expert model corresponding to each domain. Obtain the second base model, use the second base model as the student model, and set a multidimensional importance perception module and a domain gating module before the last Transformer layer of the student model; The expert models are used as teacher models to perform joint distillation training on the student models. During the joint distillation training, the multidimensional importance perception module generates a second importance score for each token in each domain dimension, and the domain gating module generates a gating probability for each token to belong to each domain. The second importance score is then fused based on the gating probability to obtain a fused importance score. The self-attention calculation and loss calculation of the student model are weighted based on the fusion importance score, and the student model is distilled based on the output of the expert model in the domain corresponding to each token to obtain the trained student model, which serves as the target large model.

[0011] According to another aspect of the present invention, a multi-domain joint fine-tuning device for a large model is provided, comprising: The data partitioning module is used to acquire training data and divide the training data into at least two domain subsets according to a preset domain partitioning standard. An expert model training module is used to obtain at least two first base models. For each first base model, the first base model is fine-tuned using the corresponding domain subset dataset. During the fine-tuning training process, the importance awareness module generates a first importance score for each token in the current processing sequence. Based on the first importance score, the self-attention calculation and loss calculation of the first base model are weighted to obtain an expert model corresponding to each domain. The model extension configuration module is used to obtain the second base model, use the second base model as the student model, and set the multidimensional importance perception module and the domain gating module before the last Transformer layer of the student model. The joint distillation processing module is used to train the student model by using each expert model as a teacher model. During the joint distillation training process, the multidimensional importance perception module generates a second importance score for each token in each domain dimension, the domain gating module generates a gating probability for each token to belong to each domain, and the second importance score is fused based on the gating probability to obtain a fused importance score. The target model generation module is used to perform weighted processing on the self-attention calculation and loss calculation of the student model based on the fusion importance score, and to perform distillation constraints on the student model based on the output of the expert model in the domain corresponding to each token, so as to obtain the trained student model as the target large model.

[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the multi-domain joint fine-tuning method for large models as described in any embodiment of the present invention.

[0013] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the multi-domain joint fine-tuning method for a large model as described in any embodiment of the present invention.

[0014] The technical solution of this invention first divides the dataset according to the characteristic differences of business data, extracts data features from different business scenarios, and trains domain expert models separately using an adaptive fine-tuning strategy, enabling them to learn the output specifications and expression styles of the corresponding scenarios. Then, using a multi-domain expert model as a teacher, joint distillation fine-tuning is performed on the Qwen3-8B base model, allowing the general model to efficiently absorb professional knowledge from various domains and take into account the answer requirements of different businesses. This solves the problems of lacking differentiated modeling in unified data mixed training and the easy confusion of multi-domain professional knowledge, achieving decoupling and efficient transfer and fusion of multi-domain professional knowledge.

[0015] Then, during the training of both the expert model and the general model, an importance weight matrix is ​​added. During the training phase, the model's master parameters and weight matrix parameters are simultaneously optimized. Weights are dynamically assigned to tokens with different semantic values ​​and incorporated into loss calculations. This strengthens the model's ability to learn core content such as numerical values, professional keywords, and regulatory sources. The importance of tokens learned during fine-tuning by domain experts is utilized to guide attention enhancement, loss weighting, and subsequent multi-expert knowledge distillation. This addresses the problems of equal-weighted token training failing to highlight key professional content, easily generating numerical and regulatory errors, and creating illusions. It achieves bidirectional reinforcement learning of core professional content, improving the accuracy of output across multiple scenarios.

[0016] Furthermore, a domain gating module is embedded during the fine-tuning stage of the general model, enabling the model to automatically identify domains. During inference, the domain weights of the gating output are dynamically adjusted to allocate the proportion of knowledge from each domain, flexibly adapting to diverse output specifications such as high-precision numerical responses, short keyword outputs, long-flow texts, and legal citations, comprehensively improving the model's scenario adaptability. This addresses the issues of dense base models lacking explicit domain identification and dynamic knowledge routing mechanisms, as well as the coupling and interference of multi-domain knowledge in the shared parameter space, achieving automatic domain differentiation of input text and soft fusion and adaptive routing of multi-domain knowledge.

[0017] Finally, when fine-tuning the general model using expert models, KL regularization constraints and dynamic domain fusion are added to the basic cross-entropy loss function. Domain classification loss is introduced to achieve supervised optimization of the model's domain recognition capability. KL divergence regularization loss constrains the model's output distribution, suppressing multi-domain knowledge confusion and generation illusion. Through the collaborative constraints and complementary optimization of multiple losses, the model's generation accuracy, domain discrimination, and multi-scenario adaptability are balanced. This addresses the problem that a single cross-entropy or fixed-weight loss cannot simultaneously address domain discrimination, knowledge consistency, and multi-scenario collaborative optimization, achieving multi-dimensional collaborative optimization of generation accuracy, domain recognition, and professional knowledge replication.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart of a multi-domain joint fine-tuning method for a large model provided in an embodiment of the present invention; Figure 2 This is a single-layer structure diagram of the fusion of the importance perception module and Transformer during the expert model training stage provided in an embodiment of the present invention. Figure 3 A flowchart of another method for multi-domain joint fine-tuning of a large model provided in an embodiment of the present invention; Figure 4 This is a diagram of the student model output layer architecture for the fine-tuning stage of the combined distillation process, provided in an embodiment of the present invention. Figure 5 This is a schematic diagram of the structure of a multi-domain joint fine-tuning device for a large model provided in an embodiment of the present invention; Figure 6 A schematic diagram of the structure of an electronic device for implementing the multi-domain joint fine-tuning method for large models in this embodiment of the invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] Figure 1 This is a flowchart illustrating a multi-domain joint fine-tuning method for a large model, provided by an embodiment of the present invention. This embodiment is applicable to scenarios where a large model needs to be applied simultaneously to multiple vertical professional fields, and where high requirements are placed on the accuracy, standardization, and domain differentiation of the responses from each field. This method can be executed by a multi-domain joint fine-tuning device for a large model, which can be implemented in hardware and / or software and can be configured in an electronic device.

[0024] like Figure 1 As shown, the method specifically includes the following steps: S110. Obtain training data and divide the training data into at least two domain subsets according to a preset domain division standard.

[0025] Among them, training data can be understood as a set of question-and-answer samples collected for various vertical business scenarios; domain division criteria refer to the data filtering rules used to distinguish different business scenarios; domain subset datasets refer to data subsets that contain only samples of a single business scenario after division.

[0026] Specifically, training data, i.e., multiple question-and-answer samples, is acquired. All question-and-answer samples are then classified and grouped according to pre-defined domain classification criteria. Each grouped sample forms an independent data subset, i.e., a domain subset. Each domain subset corresponds to a specific business scenario.

[0027] In some possible implementations, the step of acquiring training data and dividing the training data into at least two domain subsets according to a preset domain division standard includes: determining the domain division dimension based on business attributes and data characteristics; classifying the training data according to the domain division dimension to obtain at least two domain subsets, each domain subset corresponding to a different business scenario, and the training samples in each domain subset adopting a unified instruction format, and adding a domain label to each training sample to identify its domain.

[0028] Among them, business attributes refer to the business classification identifier corresponding to the industry to which the training samples belong; data features can be understood as the structural characteristics inherent in the text itself, such as the proportion of text values, the length of legal citations, text length, etc.

[0029] It should be understood that domain tags refer to identifier fields appended to samples to mark the business domain to which the sample belongs. A unified instruction format can be the Alpaca instruction storage format.

[0030] Specifically, we can analyze the industry business identifiers and text structure features inherent in all samples, and define the domain segmentation dimensions by combining these two types of information. The training samples are then classified and grouped according to these defined dimensions, generating multiple domain subsets. Furthermore, the storage structure of all samples within each domain subset is adjusted, all samples are converted to a standard instruction format, and a dedicated field is added to each sample to write the domain label.

[0031] For example, the data is categorized as follows: Based on business attributes and data characteristics, dataset D is divided into four domain sub-datasets. The classification of the four domain subsets and their key data features are shown below: Questions and answers contain a high proportion of numerical information, and the accuracy of numerical answers is required. Question-answer pairs belong to short text reasoning; Question-and-answer pairs tend to favor longer, more detailed texts. For questions and answers with numerous references, the answer should first state the name of the source document before providing the response.

[0032] Establish instruction format data: All four subsets use the Alpaca command format.

[0033] { Instruction: "How's the weather today?" "input": "", Output: "The weather is nice today; it's sunny." "domain_id": 1 } Where domain_id is the domain ID.

[0034] S120. Obtain at least two first base models. For each first base model, fine-tune the first base model using the corresponding domain subset dataset. During the fine-tuning training process, generate the first importance score of each token in the current processing sequence through the importance awareness module. Based on the first importance score, perform weighted processing on the self-attention calculation and loss calculation of the first base model to obtain the expert model corresponding to each domain.

[0035] The first base model refers to the pre-trained language base model. For example, the first base model can be Qwen3-1.7B. This embodiment does not limit the number of first base models. It can depend on the number of business domains. For example, if the number of business domains is 4, then 4 first base models can be set.

[0036] The importance-aware module can be understood as a newly added trainable linear mapping network layer, which is attached to the Transformer structure in the first basic model; Token refers to the basic semantic unit obtained after text segmentation; the first importance score refers to the numerical value that represents the importance of the corresponding token in a single domain scenario.

[0037] In this embodiment, self-attention computation refers to the computational process within the Transformer network that completes contextual semantic association matching; expert model refers to a dedicated small model that learns only single-domain knowledge and adapts to the corresponding business output specifications.

[0038] Specifically, multiple small pre-trained base models with identical structures can be retrieved, and each base model can be matched with a unique domain subset for iterative training. An importance-aware module is built within the training process. This module can read the semantic features of the text sequence and output the first importance score corresponding to each token.

[0039] Furthermore, the first importance score is incorporated into the self-attention computation process to contribute to the weights. Simultaneously, the first importance score is embedded into the loss function to adjust the gradient strength of backpropagation for different tokens. After iteration, an expert model adapted to a single business is obtained. This independent training of domain-specific expert models allows each model to independently learn the corresponding industry-specific terminology and output standards, providing a precise teacher model for the multi-domain knowledge distillation described below.

[0040] In some embodiments, the step of fine-tuning the first base model using the corresponding domain subset dataset, and generating a first importance score for each token in the current processing sequence through an importance-aware module during the fine-tuning process, and weighting the self-attention calculation and loss calculation of the first base model based on the first importance score to obtain an expert model corresponding to each domain, includes: inputting the hidden state output by the second-to-last Transformer layer in the first base model into the importance-aware module; linearly mapping the hidden state through the importance-aware module, and processing it through an activation function to obtain the first importance of each token in the current processing sequence. Scoring; In the self-attention calculation process of the first basic model, the value matrix is ​​element-wise weighted according to the first importance score of each token to obtain a weighted value matrix, and attention calculation is performed based on the weighted value matrix; The first result obtained by participating in the attention calculation with the weighted value matrix and the second result obtained by participating in the attention calculation with the original value matrix are residually fused, and the residual fusion result is input into the feedforward network layer; In the loss calculation process, the loss term corresponding to each token is weighted according to the first importance score of each token to obtain a weighted loss value, and the parameters are updated based on the weighted loss value to obtain the expert model corresponding to each domain.

[0041] In this context, the hidden state can be understood as the token semantic vector output by the Transformer network layer after processing the text, which incorporates complete contextual information and can be represented by H. The activation function can be the Sigmoid activation function, and the first importance score is used to characterize the importance of the corresponding token in its domain. The value matrix refers to the V matrix, and residual fusion refers to directly adding the results of two independent operations to preserve the complete original semantic information. The feedforward network layer refers to the MLP network inside the Transformer used for non-linear transformation of semantic features.

[0042] Specifically, the hidden text state H can be extracted from the last output of the second-to-last Transformer layer in the first basic model, and the complete hidden state H can be input into the importance perception module. The importance perception module performs linear mapping through its internal weight matrix, and then outputs the first importance score corresponding to each token after non-linear activation function operation.

[0043] It should be noted that after obtaining the above first importance score, the self-attention operation stage of the first base model can be weighted based on the first importance score, and the loss operation stage can also be weighted.

[0044] In the self-attention operation stage, the original value matrix can be scaled element-wise using the first importance score to obtain a weighted value matrix. Two independent attention operations are then performed using the weighted value matrix and the original value matrix, respectively. The two sets of operation results are then directly added to obtain the fused features, which are then fed into the feedforward network layer for further operation.

[0045] During the loss calculation stage, the loss term corresponding to the high-criticality token can be amplified based on the first importance score, and the model network parameters are updated in reverse according to the weighted loss value. After the iteration is completed, the corresponding domain expert model is output.

[0046] In this way, keyword weights are strengthened simultaneously in the two stages of attention forward computation and loss backward gradient. Under the dual-channel constraint, the learning accuracy of professional content can be improved, and the ability to express general language can be preserved through the residual structure, so that there will be no problem of stiff and incoherent sentences.

[0047] For example, Qwen3-1.7B is used as the base model for domain expert fine-tuning, such as... Figure 2 The diagram shown is a single-layer structure diagram of the importance perception module and Transformer fusion in the expert model training stage provided by an embodiment of the present invention.

[0048] An Importance Aware Module (IMP) is added to the Transformer output layer of the domain expert model. This module consists of a trainable linear mapping layer. This module is used to generate token-level importance scores based on hidden states. During training, the importance scores are used in the loss function calculation and jointly optimized with the model parameters. After training, the importance awareness module can learn the importance features of domain keywords, professional terms, legal provisions, and numerical parameters, and serve as an important basis for subsequent multi-expert knowledge fusion and domain routing.

[0049] The importance-aware module receives the hidden state H from the output of the penultimate Transformer layer as input, and generates a token-level importance vector through linear mapping and the Sigmoid activation function. in Indicates Sigmoid activation. This indicates the importance of the t-th token.

[0050] During the attention calculation phase, the V matrix of self-attention is enhanced. Then, an attention operation is performed, which directly amplifies or reduces the contribution of the token to the output according to its importance. To avoid parameter explosion and information loss, and to prevent the loss of model information, a residual network structure is added before entering the MLP layer. Loss function: Basic importance cross-loss function: In practical applications, the importance cross loss function can be used, and when calculating the loss, the importance weight matrix is ​​incorporated into the cross-entropy loss function. in, , This is the importance enhancement factor. An importance enhancement factor is introduced into the original cross-entropy loss function. When the token is highly important, its corresponding loss is amplified, prompting the model to prioritize learning domain keywords, technical terms, legal provisions, and key numerical information; when the token is less important, the loss weight is close to the original cross-entropy loss, achieving weakened learning of general vocabulary and stop words.

[0051] In some possible implementations, the step of weighting the loss term corresponding to each token based on the first importance score of each token to obtain a weighted loss value during the loss calculation process includes: obtaining manually labeled information of training samples in the domain subset, wherein the manually labeled information is used to indicate whether each token belongs to a preset key content category; determining the composite weight of each token based on the first importance score of each token and the manually labeled information; and weighting the cross-entropy loss term corresponding to each token based on the composite weight of each token to obtain a weighted loss value.

[0052] Among them, manually labeled information refers to the label information attached to the token after manually processing the samples; key content categories refer to the text content categories that require 100% accurate output in business scenarios, such as numerical content and legal citation content; composite weight refers to the comprehensive weighting coefficient obtained by fusing the first importance score of the model's autonomous learning with the manual labeling; and cross-entropy loss term refers to the basic loss term used in the training of the Sahib basic large model to constrain the accuracy of text generation.

[0053] Specifically, pre-compiled manual annotations can be read from all samples within the domain subset, with the annotation information distinguishing whether each token belongs to the key content category. The primary importance score and manual annotations corresponding to a single token are then jointly calculated to generate a composite weight specific to that token. This composite weight is applied to the cross-entropy loss component corresponding to that token, amplifying the penalty for loss related to key content, and the globally weighted loss value is calculated.

[0054] The beneficial effect of this is that it can combine the model's autonomous recognition ability with the dual constraints of human prior annotation to make up for the omissions in the model's autonomous recognition of key values ​​and legal provisions, thereby reducing the probability of errors in numerical values ​​and legal citations from the training source.

[0055] Optionally, the key content categories include numerical content and legal citation content; the manually labeled information indicates whether each token belongs to the key content category through token tags, wherein the legal citation content is wrapped and labeled using preset start and end tags.

[0056] Numerical content can be understood as various quantities, locations, and equipment measurement figures contained in the sample answers; regulatory citation content refers to the names, publication years, and original texts of laws and regulations recorded in the sample; preset start and end markers refer to paired special text identifiers, such as those using... <ref> and< / ref> As paired markers.

[0057] Specifically, the key content can be divided into two categories: one is various business measurement values, and the other is authoritative legal citations. When processing sample text, paired special tags are used to completely wrap all legal citations, and token tags are used to distinguish ordinary text from key legal texts. Numerical content is assigned a separate exclusive tag.

[0058] For example, the numerical and filename augmentation losses are as follows: Numerical Augmentation: Relying on the model to autonomously identify important content makes it difficult to accurately pinpoint all key numbers. First, all training question-and-answer data in this domain is manually annotated, adding numerical masks. Numerical words in the responses are differentiated one by one. Key values ​​and units such as the number of people, vehicles, and locations are uniformly masked as 1, while ordinary text is masked as 0. This provides the model with clear learning references, telling it which content must be accurately memorized as core values.

[0059] File name enhancement: Due to variations in file name length, special tags are used to wrap the text from which the file name originates, thus enhancing the annotation. For example: Q: What are the bases for local departments at all levels to formulate emergency response plans for sudden incidents? "A": " <ref> According to Article 17 of Chapter 2, Prevention and Emergency Preparedness, of the 2007 "Emergency Response Law":< / ref> Local authorities at all levels shall formulate corresponding emergency response plans for sudden incidents based on relevant laws, regulations, and rules, as well as the actual conditions of their respective regions. Unified use <ref>The system marks the source text of regulations in the answer package, including the publication date, full name of the regulation, policy list, and authoritative basis. Unmarked text is considered general explanatory or procedural. By marking the beginning and end of the text, the system flexibly adapts to regulatory source texts of varying lengths, accurately distinguishing between authoritative content requiring 100% replication and general descriptive content, providing the model with stable and accurate learning guidance. <ref>The content in the tag team is marked with a mask of 1, and the rest of the content is marked with a mask of 0.

[0060] Example mask: [0,1,1,1,1,1,1,1,0,0,0,0,0,0] In the model training error loss calculation stage, a double-weighted loss calculation is performed, combining the word importance scores learned autonomously by the model and the numerical mask labels manually annotated, specifically enhancing the learning effect of digital content. The improved loss calculation formula is as follows: These are manually labeled numerical markers. It is a learning weight specifically designed to amplify specific content.

[0061] S130. Obtain the second basic model, use the second basic model as the student model, and set a multidimensional importance perception module and a domain gating module before the last Transformer layer of the student model.

[0062] Among them, the second base model refers to a general pre-trained base model with more parameters than the first base model. The second base model can be Qwen3-8B; the multi-dimensional importance perception module refers to an importance perception network that is extended to a multi-channel output, with each channel corresponding to an independent business domain; the domain gating module is a newly added trainable linear network in parallel. The domain gating module is used to output the probability of the token belonging to each domain.

[0063] Specifically, a large-capacity general pre-trained pedestal model is used as the student model. A multidimensional importance perception module and a domain gating module are set before the last Transformer encoding layer of the student model. The multidimensional importance perception module and the domain gating module can synchronously receive the hidden state output by the penultimate Transformer encoding layer.

[0064] S140. The expert models are used as teacher models to perform joint distillation training on the student models. During the joint distillation training, the multidimensional importance perception module generates a second importance score for each token in each domain dimension, and the domain gating module generates a gating probability for each token to belong to each domain. The second importance score is fused based on the gating probability to obtain a fused importance score.

[0065] Among them, the teacher model refers to the expert models of each domain that have been trained in the previous text; the second importance score refers to the independent importance value of a single token in each business domain dimension; the gating probability refers to the probability value of a token belonging to each business domain output by the domain gating module; and the fusion importance score refers to the single token importance value that is finally used for global weighting after combining the weights of multiple domains and normalizing the results.

[0066] Specifically, all domain expert models are used as supervision sources, and student models are trained using a complete mixed-domain dataset. During training, the multidimensional importance perception module outputs the second importance score for each token across all domains, while the domain gating module simultaneously outputs the gating probability of each token belonging to its respective domain. The second importance score of the corresponding dimension is weighted using the gating probability, and after normalization, a unique fused importance score is obtained.

[0067] S150. The self-attention calculation and loss calculation of the student model are weighted based on the fusion importance score, and the student model is distilled based on the output of the expert model in the domain corresponding to each token to obtain the trained student model as the target large model.

[0068] Among them, distillation constraint refers to using the semantic distribution of the teacher model output to supervise the distribution of the student model output; target large model refers to the final large model that has completed multi-domain joint fine-tuning and can be directly deployed to multiple vertical business scenarios.

[0069] Specifically, the importance score of the fusion is substituted into the self-attention operation and global loss operation of the student model to achieve double weighting. Then, the domain label corresponding to each training sample is read and matched with the output semantic distribution of the corresponding domain expert model; the distribution difference is used to constrain the parameter updates of the student model. After the complete iterative process is completed, the converged student model is output as the target large model.

[0070] The technical solution of this invention first divides the dataset according to the characteristic differences of business data, extracts data features from different business scenarios, and trains domain expert models separately using an adaptive fine-tuning strategy, enabling them to learn the output specifications and expression styles of the corresponding scenarios. Then, using a multi-domain expert model as a teacher, joint distillation fine-tuning is performed on the Qwen3-8B base model, allowing the general model to efficiently absorb professional knowledge from various domains and take into account the answer requirements of different businesses. This solves the problems of lacking differentiated modeling in unified data mixed training and the easy confusion of multi-domain professional knowledge, achieving decoupling and efficient transfer and fusion of multi-domain professional knowledge.

[0071] Then, during the training of both the expert model and the general model, an importance weight matrix is ​​added. During the training phase, the model's master parameters and weight matrix parameters are simultaneously optimized. Weights are dynamically assigned to tokens with different semantic values ​​and incorporated into loss calculations. This strengthens the model's ability to learn core content such as numerical values, professional keywords, and regulatory sources. The importance of tokens learned during fine-tuning by domain experts is utilized to guide attention enhancement, loss weighting, and subsequent multi-expert knowledge distillation. This addresses the problems of equal-weighted token training failing to highlight key professional content, easily generating numerical and regulatory errors, and creating illusions. It achieves bidirectional reinforcement learning of core professional content, improving the accuracy of output across multiple scenarios.

[0072] Furthermore, a domain gating module is embedded during the fine-tuning stage of the general model, enabling the model to automatically identify domains. During inference, the domain weights of the gating output are dynamically adjusted to allocate the proportion of knowledge from each domain, flexibly adapting to diverse output specifications such as high-precision numerical responses, short keyword outputs, long-flow texts, and legal citations, comprehensively improving the model's scenario adaptability. This addresses the issues of dense base models lacking explicit domain identification and dynamic knowledge routing mechanisms, as well as the coupling and interference of multi-domain knowledge in the shared parameter space, achieving automatic domain differentiation of input text and soft fusion and adaptive routing of multi-domain knowledge.

[0073] Finally, when fine-tuning the general model using expert models, KL regularization constraints and dynamic domain fusion are added to the basic cross-entropy loss function. Domain classification loss is introduced to achieve supervised optimization of the model's domain recognition capability. KL divergence regularization loss constrains the model's output distribution, suppressing multi-domain knowledge confusion and generation illusion. Through the collaborative constraints and complementary optimization of multiple losses, the model's generation accuracy, domain discrimination, and multi-scenario adaptability are balanced. This addresses the problem that a single cross-entropy or fixed-weight loss cannot simultaneously address domain discrimination, knowledge consistency, and multi-scenario collaborative optimization, achieving multi-dimensional collaborative optimization of generation accuracy, domain recognition, and professional knowledge replication.

[0074] Figure 3 This is a flowchart illustrating another method for multi-domain joint fine-tuning of a large model provided by an embodiment of the present invention. Based on the above embodiments, this embodiment further optimizes the joint distillation learning process. Specific implementation details can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 3 As shown, the method specifically includes the following steps: S310. Obtain training data and divide the training data into at least two domain subsets according to a preset domain division standard.

[0075] S320. Obtain at least two first base models. For each first base model, fine-tune the first base model using the corresponding domain subset dataset. During the fine-tuning training process, generate the first importance score of each token in the current processing sequence through the importance awareness module. Based on the first importance score, perform weighted processing on the self-attention calculation and loss calculation of the first base model to obtain the expert model corresponding to each domain.

[0076] S330. Obtain a second basic model, wherein the number of parameters of the second basic model is greater than the number of parameters of the first basic model.

[0077] S340. Set the multidimensional importance perception module before the last Transformer layer of the second basic model. The multidimensional importance perception module is used to receive the hidden state output by the penultimate Transformer layer and generate the second importance score of each token in each domain dimension.

[0078] S350. The domain gating module is set up in parallel before the last Transformer layer of the second basic model. The domain gating module is used to receive the hidden state output by the penultimate Transformer layer and generate the gating probability of each token belonging to each domain.

[0079] Among them, the number of channels of the multidimensional importance perception module matches the total number of business domains; the domain gating module and the multidimensional importance perception module are deployed in parallel, and the two types of modules share the hidden state input of the same layer of Transformer output.

[0080] Specifically, a high-capacity pre-trained pedestal model with a higher total number of network parameters than the first base model is selected as the student model. A multi-dimensional importance-aware module is built before the last Transformer encoding layer of the student model. This module can read the hidden state output by the penultimate Transformer encoding layer and output the second importance score of each token across all business domains. Additionally, a domain gating module is deployed in parallel at the same location. This module can synchronously read the hidden state output by the penultimate Transformer encoding layer and output the gating probability of each token belonging to each business domain.

[0081] S360. The expert models are used as teacher models to perform joint distillation training on the student models. During the joint distillation training, the multidimensional importance perception module generates a second importance score for each token in each domain dimension, and the domain gating module generates a gating probability for each token to belong to each domain. The second importance score is fused based on the gating probability to obtain a fused importance score.

[0082] In some possible implementations, the step of fusing the second importance score based on the gating probability to obtain the fused importance score includes: for each token in the current processing sequence, obtaining the second importance score of the token in each domain dimension to form a multidimensional importance score vector of the token, and obtaining the gating probability of the token belonging to each domain generated by the domain gating module; multiplying each probability value in the gating probability with the score value of the corresponding dimension in the multidimensional importance score vector by weight to obtain the weighted score value of each dimension; summing the score values ​​of each dimension and dividing by the sum of the probability values ​​to obtain the fused importance score of the token.

[0083] Among them, the multidimensional importance score vector refers to the vector formed by splicing the second importance scores of all business areas corresponding to a single token; the weighted multiplication refers to multiplying the gating probability with the corresponding second importance score in each dimension; the normalization operation refers to summing the weighted multidimensional scores and dividing by the sum of all gating probability values ​​to obtain a unique standardized score.

[0084] Specifically, for each token in the text sequence, the second importance score corresponding to that token across all domains is extracted and concatenated to form a multi-dimensional importance score vector. Simultaneously, all domain gating probabilities output by the domain gating module are read. Each set of gating probabilities is multiplied by the second importance score of the same dimension within the vector. The sum of all dimension multiplication results is then calculated and divided by the sum of all gating probabilities to obtain the unique fusion importance score for that token.

[0085] It should also be noted that the normalization operation here constrains the range of importance scores to avoid weight overflow when multiple domains are activated simultaneously, thus ensuring the stability of the self-attention operation values.

[0086] For example, the fine-tuning process of combined distillation is as follows: Figure 4 This is a diagram of the student model output layer architecture for the joint distillation fine-tuning stage provided in an embodiment of the present invention. This model architecture uses Qwen3-8B as the student model for joint distillation fine-tuning, and the training set is the full dataset. Four domain expert models from Qwen3-1.7B are used as teacher models to train the student model. These four domain expert models also serve as teacher models, jointly distilling and training the Qwen3-8B student model to achieve dynamic fusion and transfer of knowledge across multiple domains.

[0087] The Qwen3-8B student model adopts the same architecture as the expert model: an importance-aware module is added, the original Transformer backbone parameters are fine-tuned using full parameters, and the importance-aware module uses newly added randomly initialized parameters. The training phase and the base parameters are updated synchronously and jointly.

[0088] The joint distillation fine-tuning of the student model extends the importance perception module on the expert model architecture, adding domain-based importance weights. The original one-dimensional domain importance of students is expanded to K dimensions, and K can be changed according to the number of expert models or business classification.

[0089] The hidden state H, which is the output of the penultimate Transformer layer, is used as the input to the last Transformer layer. Token-level importance vectors in different domains are generated through linear mapping and the Sigmoid activation function. in It is the importance vector representation of the token in the K-dimensional domain.

[0090] Simultaneously, a domain gating module is added alongside the importance weight matrix, enabling the large model to learn problem classification. To adapt to multi-domain cross-scenarios and avoid gradient conflicts caused by hard selection, this system employs a Sigmoid-based non-competitive gating and continuously weighted soft fusion mechanism. For each token, the domain gating module calculates its independent activation probability in each K-dimensional domain through linear mapping and the Sigmoid activation function: in, Indicates Sigmoid activation. Let be the trainable weight matrix for the domain gating module. Let t represent the t-th token, where . This represents the independent activation probability of a token belonging to the k-th domain, and is a domain gating vector. The kth element.

[0091] After obtaining the domain gating probability vector Then, the probability is used as a weight for the multidimensional importance vector. Continuous weighted soft fusion is performed. Furthermore, to prevent excessive weight accumulation during multi-domain co-activation, which could lead to unstable attention values, a normalization improvement operator is introduced to calculate the final fusion importance vector. in To prevent division by a zero smoothing factor, this operator ensures... The final fusion importance score is strictly constrained to... Between these parameters, and when all channels are inactive (i.e., general words and stop words), it can naturally converge to a silent state (close to 0).

[0092] Obtain the importance vector Then, the V matrix of self-attention is enhanced. Then, an attention operation is performed, which directly amplifies or reduces the contribution of the token to the output according to its importance. To avoid parameter explosion and information loss, and to prevent the loss of model information, a residual network structure is added before entering the MLP layer. S370. The self-attention calculation and loss calculation of the student model are weighted based on the fusion importance score, and the student model is distilled based on the output of the expert model in the domain corresponding to each token to obtain the trained student model as the target large model.

[0093] In some possible implementations, the step of using each of the expert models as teacher models to jointly distill the student model includes: during the joint distillation training process, determining the target expert model corresponding to the domain label from each of the expert models based on the domain label of the training samples; calculating the divergence loss between the probability distribution output by the student model and the probability distribution output by the target expert model; generating the domain prediction probability of each token based on the hidden state of each token in the training samples through the domain gating module, and weighting the domain prediction loss according to the fusion importance score of each token to obtain the domain classification loss; weighting and summing the cross-entropy loss, the divergence loss, and the domain classification loss based on the fusion importance score to obtain a composite loss value; and jointly updating the parameters of the student model based on the composite loss value to obtain the trained student model.

[0094] Here, the target expert model refers to a single domain expert model that matches the domain labels of the current training samples. Divergence loss can be understood as a loss term that measures the difference in semantic distribution between the student model and the teacher model's outputs. Domain prediction probability refers to the predicted probability of a single token output by the domain gating module belonging to each domain; domain classification loss refers to the loss term used to supervise the domain recognition accuracy of the gating module; and the composite loss value refers to the total global loss obtained by weighted summing of the basic weighted cross-entropy loss, divergence loss, and domain classification loss.

[0095] Specifically, within the joint distillation training process, the domain label bound to a single training sample is read, and the corresponding domain-trained expert model is used as the target expert model. The divergence loss between the semantic distribution output by the student model and the semantic distribution output by the target expert model is calculated.

[0096] Furthermore, the hidden state corresponding to each token is read and sent to the domain gating module to generate the domain prediction probability of that token in each domain. The loss component corresponding to the domain prediction probability is weighted using the fusion importance score to generate the domain classification loss. The cross-entropy loss, divergence loss, and domain classification loss after fusion importance weighting are summed according to preset weights to obtain the global composite loss value. All network parameters of the student model are updated synchronously based on the composite loss value.

[0097] In some possible implementations, calculating the divergence loss between the probability distribution output by the student model and the probability distribution output by the target expert model includes: obtaining a preset temperature coefficient; smoothing the logic values ​​output by the student model and the logic values ​​output by the target expert model according to the temperature coefficient to obtain smoothed student probability distributions and smoothed target probability distributions; and calculating the KL divergence between the smoothed student probability distributions and the smoothed target probability distributions as the divergence loss.

[0098] Among them, the temperature coefficient refers to the hyperparameter used to smooth the semantic distribution of the model output; the logistic value refers to the original predicted value of the model output before normalization; the smoothed probability distribution refers to the softened output distribution obtained by dividing the logistic value by the temperature coefficient and then normalizing it; and the KL divergence refers to the calculation index that quantifies the degree of difference between the two probability distributions.

[0099] Specifically, a pre-set temperature coefficient can be retrieved to scale the original logic values ​​output by the student model and the target expert model, respectively, and then normalized to generate two sets of smoothed probability distributions. The KL divergence value between the two sets of smoothed distributions is calculated, and this value is used as the divergence loss for the distillation process.

[0100] For example, the loss function is as follows: The overall loss function of combined distillation consists of three weighted parts: To adjust the hyperparameters for the proportion of various losses, and to meet the requirements... Or adjust independently based on experimental performance.

[0101] The loss formulas for each sub-item are defined as follows: Importance of basic losses The joint fine-tuning phase uses the full dataset for tuning. The joint model receives samples from four domains simultaneously, but still maintains strict requirements regarding the source of numerical values ​​and file names. Therefore, an adaptive, smooth transition loss function is designed for the joint fine-tuning phase, based on the student model to predict probabilities. and dynamically selected token importance weights : : mask weight coefficient (between 0 and 1), e: current training epoch. This indicates from which round the decay begins (e.g., round 20). Total number of training rounds : Attenuation rate.

[0102] KL Regularized Loss Based on the real domain labels of the training samples Load the corresponding expert teacher model Calculate the probability distribution of the student model output. Probability distribution of teacher expert model output KL divergence between (introducing temperature coefficient) (with smooth distribution) in: The Logits output by the model. This is the distillation temperature, used to balance the information content of the soft label.

[0103] Domain loss ( The predictions of the gating module are constrained by the vocabulary size to approximate the true domain labels of the samples. The cross-entropy loss is calculated between the domain prediction probability and the true domain label for each token in the sequence. Using... The domain loss is weighted as a factor: in This is the index of the real domain to which the current training sample belongs. Only when a token has extremely high domain specificity (high importance score, such as "wet alarm valve" or "water pump connector") is the strongly supervised gating network used to learn its domain affiliation; while for general words and weak domain words, they are directly "let go" with low weights or masks, allowing the gating output to have ambiguous probabilities (such as [0.4, 0.4, 0.1, 0.1]) without penalty.

[0104] For question-and-answer scenarios in different task scenarios within a vertical domain, the training set is divided from a data business perspective to form an expert training set for the vertical domain, and then the expert model is fine-tuned.

[0105] A token-level importance-aware system based on bidirectional constraints (forward attention and backward gradient) adds an importance-aware module (IMP) after the penultimate Transformer layer and before the last Transformer layer. And the activation function generates token-level importance scalars in the range [0,1]. In the forward computation, it utilizes... The value matrix of the self-attention mechanism is element-wise weighted, and a dual-channel residual design is employed. In the loss calculation, [the following is used:] The cross-entropy loss function is introduced as a weighting factor to dynamically adjust the gradient backpropagation strength of different tokens during backpropagation. This addresses the problem of insufficient attention to keywords (such as specific values ​​or normative rules) in traditional fine-tuning, and the issue that directly modifying the attention mechanism can easily lead to the loss of general semantic information and distortion of language generation.

[0106] A K-dimensional domain-gating and dynamic routing fusion architecture for multi-expert joint distillation is proposed. During the joint distillation fine-tuning stage, the one-dimensional importance-aware module of the student model is expanded into a K-dimensional matrix, with each dimension corresponding to a specific expert domain. Parallel domain-gating modules are introduced. It calculates the probability distribution of the current token's sub-sector in real time and extracts the importance score for the corresponding dimension. Dynamically intervene in the self-attention calculation of the current token. Design a composite loss function: Constrain the basic generative capabilities of the student model; dynamically call the corresponding expert model based on the actual Domain_id of the sample to perform KL divergence constraints. This enables precise transfer of specialized knowledge; through The classification accuracy of the constraint gating module.

[0107] The technical solution of this invention first divides the dataset according to the characteristic differences of business data, extracts data features from different business scenarios, and trains domain expert models separately using an adaptive fine-tuning strategy, enabling them to learn the output specifications and expression styles of the corresponding scenarios. Then, using a multi-domain expert model as a teacher, joint distillation fine-tuning is performed on the Qwen3-8B base model, allowing the general model to efficiently absorb professional knowledge from various domains and take into account the answer requirements of different businesses. This solves the problems of lacking differentiated modeling in unified data mixed training and the easy confusion of multi-domain professional knowledge, achieving decoupling and efficient transfer and fusion of multi-domain professional knowledge.

[0108] Then, during the training of both the expert model and the general model, an importance weight matrix is ​​added. During the training phase, the model's master parameters and weight matrix parameters are simultaneously optimized. Weights are dynamically assigned to tokens with different semantic values ​​and incorporated into loss calculations. This strengthens the model's ability to learn core content such as numerical values, professional keywords, and regulatory sources. The importance of tokens learned during fine-tuning by domain experts is utilized to guide attention enhancement, loss weighting, and subsequent multi-expert knowledge distillation. This addresses the problems of equal-weighted token training failing to highlight key professional content, easily generating numerical and regulatory errors, and creating illusions. It achieves bidirectional reinforcement learning of core professional content, improving the accuracy of output across multiple scenarios.

[0109] Furthermore, a domain gating module is embedded during the fine-tuning stage of the general model, enabling the model to automatically identify domains. During inference, the domain weights of the gating output are dynamically adjusted to allocate the proportion of knowledge from each domain, flexibly adapting to diverse output specifications such as high-precision numerical responses, short keyword outputs, long-flow texts, and legal citations, comprehensively improving the model's scenario adaptability. This addresses the issues of dense base models lacking explicit domain identification and dynamic knowledge routing mechanisms, as well as the coupling and interference of multi-domain knowledge in the shared parameter space, achieving automatic domain differentiation of input text and soft fusion and adaptive routing of multi-domain knowledge.

[0110] Finally, when fine-tuning the general model using expert models, KL regularization constraints and dynamic domain fusion are added to the basic cross-entropy loss function. Domain classification loss is introduced to achieve supervised optimization of the model's domain recognition capability. KL divergence regularization loss constrains the model's output distribution, suppressing multi-domain knowledge confusion and generation illusion. Through the collaborative constraints and complementary optimization of multiple losses, the model's generation accuracy, domain discrimination, and multi-scenario adaptability are balanced. This addresses the problem that a single cross-entropy or fixed-weight loss cannot simultaneously address domain discrimination, knowledge consistency, and multi-scenario collaborative optimization, achieving multi-dimensional collaborative optimization of generation accuracy, domain recognition, and professional knowledge replication.

[0111] Figure 5 This is a schematic diagram of the structure of a multi-domain joint fine-tuning device for a large model provided in an embodiment of the present invention. Figure 5 As shown, the device includes: The data partitioning module 510 is used to acquire training data and divide the training data into at least two domain subsets according to a preset domain partitioning standard. The expert model training module 520 is used to obtain at least two first base models. For each first base model, the first base model is fine-tuned using the corresponding domain subset dataset. During the fine-tuning training process, the importance perception module generates the first importance score of each token in the current processing sequence. Based on the first importance score, the self-attention calculation and loss calculation of the first base model are weighted to obtain the expert model corresponding to each domain. The model extension configuration module 530 is used to obtain a second basic model, use the second basic model as a student model, and set a multidimensional importance perception module and a domain gating module before the last Transformer layer of the student model. The joint distillation processing module 540 is used to perform joint distillation training on the student model using each of the expert models as teacher models. During the joint distillation training process, the multidimensional importance perception module generates a second importance score for each of the tokens in each domain dimension, the domain gating module generates a gating probability for each of the tokens to belong to each domain, and the second importance score is fused based on the gating probability to obtain a fused importance score. The target model generation module 550 is used to perform weighted processing on the self-attention calculation and loss calculation of the student model based on the fusion importance score, and to perform distillation constraints on the student model based on the output of the expert model in the domain corresponding to each token, so as to obtain the trained student model as the target large model.

[0112] In some possible implementations, the data partitioning module 510 includes: The dimension determination submodule is used to determine the domain division dimensions based on business attributes and data characteristics; The sample classification submodule is used to classify the training data according to the domain division dimension to obtain at least two domain sub-datasets. Each domain sub-dataset corresponds to a different business scenario, and the training samples in each domain sub-dataset adopt a unified instruction format. A domain label is added to each training sample to identify its domain.

[0113] In some possible implementations, the expert model training module 520 includes: The feature input submodule is used to input the hidden state output by the second-to-last Transformer layer in the first basic model into the importance perception module; The first score generation submodule is used to perform a linear mapping of the hidden state through the importance perception module and process it through an activation function to obtain the first importance score of each token in the current processing sequence. The first importance score is used to characterize the importance of the corresponding token in its respective domain. The attention weighting submodule is used to perform element-wise weighting of the value matrix according to the first importance score of each token during the self-attention calculation process of the first basic model, to obtain a weighted value matrix, and to perform attention calculation based on the weighted value matrix. The residual fusion submodule is used to perform residual fusion between the first result obtained by participating the weighted value matrix in attention calculation and the second result obtained by participating the original value matrix in attention calculation, and input the residual fusion result into the feedforward network layer; The parameter update submodule is used to perform weighted processing on the loss items corresponding to each token according to the first importance score of each token during the loss calculation process, to obtain the weighted loss value, and to update the parameters based on the weighted loss value to obtain the expert model corresponding to each domain.

[0114] In some possible implementations, the parameter update submodule includes: The annotation reading unit is used to obtain manually annotated information for training samples in the domain subset dataset. The manually annotated information is used to indicate whether each token belongs to a preset key content category. The composite weight calculation unit is used to determine the composite weight of each token based on the first importance score of each token and the manually labeled information. The loss weighting unit is used to weight the cross-entropy loss term corresponding to each token based on the composite weight of each token, so as to obtain the weighted loss value.

[0115] In some possible implementations, the key content categories include numerical content and legal citation content; The manually labeled information indicates whether each token belongs to the key content category through token tags. The legal citation content is wrapped and labeled using preset start and end tags.

[0116] In some possible implementations, the model extension configuration module 530 includes: The base selection submodule is used to obtain a second base model, the number of parameters of the second base model is greater than the number of parameters of the first base model; The multidimensional perception deployment submodule is used to set up the multidimensional importance perception module before the last Transformer layer of the second base model. The multidimensional importance perception module is used to receive the hidden state output by the penultimate Transformer layer and generate the second importance score of each token in each domain dimension. The gating deployment submodule is used to set up the domain gating module in parallel before the last Transformer layer of the second basic model. The domain gating module is used to receive the hidden state output by the penultimate Transformer layer and generate the gating probability of each token belonging to each domain.

[0117] In some possible implementations, the combined distillation processing module 540 includes: The multidimensional scoring vector construction submodule is used to obtain the second importance score of each token in each domain dimension for each token in the current processing sequence, form the multidimensional importance score vector of the token, and obtain the gate probability of the token belonging to each domain generated by the domain gate module; The dimension weighting submodule is used to perform weighted multiplication of each probability value in the gated probability with the corresponding dimension score value in the multidimensional importance score vector to obtain the weighted score value of each dimension. The normalization fusion submodule is used to sum the scores of each dimension and divide them by the sum of the probability values ​​to obtain the fusion importance score of the token.

[0118] In some possible implementations, the combined distillation processing module 540 further includes: The teacher matching submodule is used to determine the target expert model corresponding to the domain label from each of the expert models during the joint distillation training process, based on the domain label of the training sample. The divergence loss calculation submodule is used to calculate the divergence loss between the probability distribution output by the student model and the probability distribution output by the target expert model. The domain loss weighting submodule is used to generate the domain prediction probability of each token based on the hidden state of each token in the training samples through the domain gating module, and to weight the domain prediction loss according to the fusion importance score of each token to obtain the domain classification loss. The composite loss synthesis submodule is used to perform a weighted summation of the cross-entropy loss, the divergence loss, and the domain classification loss, which are weighted based on the fusion importance score, to obtain a composite loss value. The parameter synchronization update submodule is used to jointly update the parameters of the student model based on the composite loss value to obtain the trained student model.

[0119] In some possible implementations, the divergence loss calculation submodule includes: A distribution smoothing unit is used to obtain a preset temperature coefficient and smooth the logic values ​​output by the student model and the target expert model respectively according to the temperature coefficient to obtain the smoothed student probability distribution and the smoothed target probability distribution. The KL divergence calculation unit is used to calculate the KL divergence between the smoothed student probability distribution and the smoothed target probability distribution, as the divergence loss.

[0120] The multi-domain joint fine-tuning device for large models provided in the embodiments of the present invention can execute the multi-domain joint fine-tuning method for large models provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0121] Figure 6 This is a schematic diagram of an electronic device for implementing the multi-domain joint fine-tuning method for a large model according to embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptops, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0122] like Figure 6 As shown, the electronic device 60 includes at least one processor 61 and a memory, such as a read-only memory (ROM) 62 and a random access memory (RAM) 63, communicatively connected to the at least one processor 61. The memory stores computer programs executable by the at least one processor. The processor 61 can perform various appropriate actions and processes based on the computer program stored in the ROM 62 or loaded from storage unit 68 into the RAM 63. The RAM 63 can also store various programs and data required for the operation of the electronic device 60. The processor 61, ROM 62, and RAM 63 are interconnected via a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.

[0123] Multiple components in electronic device 60 are connected to I / O interface 65, including: input unit 66, such as keyboard, mouse, etc.; output unit 67, such as various types of monitors, speakers, etc.; storage unit 68, such as disk, optical disk, etc.; and communication unit 69, such as network card, modem, wireless transceiver, etc. Communication unit 69 allows electronic device 60 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0124] Processor 61 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 61 performs the various methods and processes described above, such as multi-domain joint fine-tuning methods for large models.

[0125] In some embodiments, the multi-domain joint fine-tuning method for large models can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 68. In some embodiments, part or all of the computer program can be loaded and / or mounted on electronic device 60 via ROM 62 and / or communication unit 69. When the computer program is loaded into RAM 63 and executed by processor 61, one or more steps of the multi-domain joint fine-tuning method for large models described above can be performed. Alternatively, in other embodiments, processor 61 can be configured to perform the multi-domain joint fine-tuning method for large models by any other suitable means (e.g., by means of firmware).

[0126] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific integrated circuits (ASICs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0130] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0131] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0132] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0133] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.< / ref> < / ref>

Claims

1. A multi-domain joint fine-tuning method for a large model, characterized in that, include: Acquire training data and divide the training data into at least two domain subsets according to a preset domain division standard; At least two first base models are obtained. For each first base model, the first base model is fine-tuned using the corresponding domain subset dataset. During the fine-tuning training process, the first importance score of each token in the current processing sequence is generated through the importance awareness module. The self-attention calculation and loss calculation of the first base model are weighted based on the first importance score to obtain the expert model corresponding to each domain. Obtain the second basic model, use the second basic model as the student model, and set a multidimensional importance perception module and a domain gating module in the last Transformer layer of the student model; The expert models are used as teacher models to perform joint distillation training on the student models. During the joint distillation training, the multidimensional importance perception module generates a second importance score for each token in each domain dimension, and the domain gating module generates a gating probability for each token to belong to each domain. The second importance score is then fused based on the gating probability to obtain a fused importance score. The self-attention calculation and loss calculation of the student model are weighted based on the fusion importance score, and the student model is distilled based on the output of the expert model in the domain corresponding to each token to obtain the trained student model, which serves as the target large model.

2. The method according to claim 1, characterized in that, The acquisition of training data involves dividing the training data into at least two domain subsets according to a preset domain segmentation standard, including: Determine the domain segmentation dimensions based on business attributes and data characteristics; The training data is classified according to the domain division dimension to obtain at least two domain sub-datasets. Each domain sub-dataset corresponds to a different business scenario, and the training samples in each domain sub-dataset adopt a unified instruction format. A domain label is added to each training sample to identify its domain.

3. The method according to claim 1, characterized in that, The first base model is fine-tuned using the corresponding domain subset dataset. During the fine-tuning process, an importance-aware module generates a first importance score for each token in the current processing sequence. Based on the first importance score, the self-attention calculation and loss calculation of the first base model are weighted to obtain an expert model corresponding to each domain, including: The hidden state output by the second-to-last Transformer layer in the first basic model is input into the importance perception module; The importance perception module performs a linear mapping on the hidden state and processes it through an activation function to obtain the first importance score of each token in the current processing sequence. The first importance score is used to characterize the importance of the corresponding token in its respective domain. In the self-attention calculation process of the first basic model, the value matrix is ​​element-wise weighted according to the first importance score of each token to obtain a weighted value matrix, and attention calculation is performed based on the weighted value matrix. The first result obtained by participating the weighted value matrix in the attention calculation is fused with the second result obtained by participating the original value matrix in the attention calculation. The residual fusion result is then input into the feedforward network layer. During the loss calculation process, the loss items corresponding to each token are weighted according to the first importance score of each token to obtain a weighted loss value. The parameters are then updated based on the weighted loss value to obtain an expert model corresponding to each domain.

4. The method according to claim 3, characterized in that, In the loss calculation process, the loss items corresponding to each token are weighted according to the first importance score of each token to obtain a weighted loss value, including: Obtain manually labeled information for training samples in the domain subset dataset, wherein the manually labeled information is used to indicate whether each token belongs to a preset key content category; The composite weight of each token is determined based on the first importance score of each token and the manually labeled information. The cross-entropy loss term corresponding to each token is weighted based on the composite weight of each token to obtain the weighted loss value.

5. The method according to claim 4, characterized in that, The key content categories include numerical content and legal citation content; The manually labeled information indicates whether each token belongs to the key content category through token tags. The legal citation content is wrapped and labeled using preset start and end tags.

6. The method according to claim 1, characterized in that, The process of obtaining the second base model, using it as a student model, and setting a multi-dimensional importance perception module and a domain gating module in the last Transformer layer of the student model, includes: Obtain a second base model, the number of parameters of the second base model is greater than the number of parameters of the first base model; The multidimensional importance perception module is set before the last Transformer layer of the second basic model. The multidimensional importance perception module is used to receive the hidden state output by the penultimate Transformer layer and generate the second importance score of each token in each domain dimension. The domain gating module is set up in parallel before the last Transformer layer of the second basic model. The domain gating module is used to receive the hidden state output by the penultimate Transformer layer and generate the gating probability of each token belonging to each domain.

7. The method according to claim 6, characterized in that, The process of fusing the second importance score based on the gating probability to obtain the fused importance score includes: For each token in the current processing sequence, obtain the second importance score of the token under each domain dimension, form a multidimensional importance score vector of the token, and obtain the gating probability of the token belonging to each domain generated by the domain gating module; Each probability value in the gating probability is multiplied by the corresponding dimension score value in the multidimensional importance score vector to obtain the weighted score value for each dimension. The fusion importance score of the token is obtained by summing the scores of each dimension and dividing by the sum of the probability values.

8. The method according to claim 1, characterized in that, The method of using each of the expert models as teacher models to perform joint distillation training on the student model includes: During the joint distillation training process, the target expert model corresponding to the domain label is determined from each of the expert models based on the domain label of the training samples; Calculate the divergence loss between the probability distribution output by the student model and the probability distribution output by the target expert model; The domain gating module generates the domain prediction probability of each token based on the hidden state of each token in the training samples, and weights the domain prediction loss according to the fusion importance score of each token to obtain the domain classification loss. The weighted cross-entropy loss, the divergence loss, and the domain classification loss, which are weighted based on the fusion importance score, are summed to obtain a composite loss value. The parameters of the student model are jointly updated based on the composite loss value to obtain the trained student model.

9. The method according to claim 8, characterized in that, The calculation of the divergence loss between the probability distribution output by the student model and the probability distribution output by the target expert model includes: Obtain a preset temperature coefficient, and smooth the logic values ​​output by the student model and the target expert model respectively according to the temperature coefficient to obtain the smoothed student probability distribution and the smoothed target probability distribution. The KL divergence between the smoothed student probability distribution and the smoothed target probability distribution is calculated as the divergence loss.

10. A multi-domain joint fine-tuning device for a large model, characterized in that, include: The data partitioning module is used to acquire training data and divide the training data into at least two domain subsets according to a preset domain partitioning standard. An expert model training module is used to obtain at least two first base models. For each first base model, the first base model is fine-tuned using the corresponding domain subset dataset. During the fine-tuning training process, the importance awareness module generates a first importance score for each token in the current processing sequence. Based on the first importance score, the self-attention calculation and loss calculation of the first base model are weighted to obtain an expert model corresponding to each domain. The model extension configuration module is used to obtain the second base model, use the second base model as the student model, and set the multidimensional importance perception module and the domain gating module before the last Transformer layer of the student model. The joint distillation processing module is used to train the student model by using each expert model as a teacher model. During the joint distillation training process, the multidimensional importance perception module generates a second importance score for each token in each domain dimension, the domain gating module generates a gating probability for each token to belong to each domain, and the second importance score is fused based on the gating probability to obtain a fused importance score. The target model generation module is used to perform weighted processing on the self-attention calculation and loss calculation of the student model based on the fusion importance score, and to perform distillation constraints on the student model based on the output of the expert model in the domain corresponding to each token, so as to obtain the trained student model as the target large model.

11. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the multi-domain joint fine-tuning method for a large model as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a processor to execute the multi-domain joint fine-tuning method for a large model as described in any one of claims 1-9.

Images