A domain data driven large model adaptation system and method

Abstract

The application relates to the technical field of natural language processing, in particular to a domain data driven large model adaptation system and method; the method comprises the following steps: comparing the segmentation result of a base model's word segmenter on domain corpus with a domain term library, calculating the segmentation fragmentation degree and screening target terms; generating a term semantic anchor point by encoding the definition text of the target term with the base model, comparing the overall representation of the term with the semantic anchor point to obtain the deviation direction and deviation amplitude; under the conditions of the word segmenter and weight freezing, constructing a correction mapping acting on subword embedding, and initializing the correction direction and correction strength with the deviation direction and deviation amplitude; and jointly fine-tuning the correction mapping and low-rank adaptation parameters with the domain corpus to obtain a domain adaptation model. The application realizes the rapid switching and large-scale deployment of the same base model among multiple professional fields without changing the word segmenter and the base model weight.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, specifically to a domain data-driven large model adaptation system and method. Background Technology

[0002] Existing large-scale language model adaptation schemes for specialized domains typically involve fine-tuning a base model with domain-specific corpora to adapt it to the domain-specific text. The adapted model is then used for classification and generation tasks within that domain. However, in actual specialized text processing, the model's performance on different terms within the same batch of domain text is inconsistent: terms composed of common words are mostly understood and used correctly, while some specialized terms, even after the same domain-specific fine-tuning, still exhibit significant deviations in their meaning, repeatedly leading to errors in downstream classification and generation. These errors often occur with highly specialized terms that are crucial to the processing results.

[0003] If the above problems are not effectively addressed, the more core the domain's technical terms, the more easily the model will misunderstand them. Downstream request classification may misassign key work orders to the wrong departments, and work order summaries may distort the key points of processing, making it difficult for responsible personnel to rely on the model's output to complete professional text processing, thus reducing the usability of the adapted model in the professional domain. Therefore, it is necessary to solve the problem that after adapting a large language model to a professional domain, some technical terms, even after domain-specific fine-tuning, are still misunderstood by the model, leading to continuous errors in downstream text classification and generation.

[0004] In view of this, this application proposes a domain data-driven large model adaptation system and method. Summary of the Invention

[0005] To achieve the above objectives, this application provides a domain data-driven large model adaptation system and method, the specific technical solution of which is as follows: Firstly, this application provides a domain data-driven method for adapting large models, including: The word segmentation results of the base model's word segmenter on the domain corpus are compared with the domain terminology database. The number of sub-words segmented by the hit terms and the number of times the sub-words appear in the general corpus are counted. The segmentation fragmentation is calculated, and target terms with segmentation fragmentation exceeding the preset threshold are selected. The definition text of the target term is encoded by the base model to generate semantic anchors of the term; the word representations of the target term in the domain corpus are extracted and aggregated into the overall representation of the term; the overall representation of the term is compared with the semantic anchors of the term to obtain the deviation direction and deviation magnitude. With the word segmenter and weights of the base model frozen, a correction mapping for the word embedding of the target term is constructed, and the correction direction and correction intensity of the correction mapping are initialized with the deviation direction and deviation magnitude, respectively. The domain-specific corpus is used to jointly fine-tune the correction mapping and low-rank adaptation parameters. The joint fine-tuning loss includes term-level alignment loss weighted by segmentation fragmentation degree, term-level overall representation relative to term semantic anchors, and sentence-level language modeling loss, thus obtaining the domain adaptation model.

[0006] Preferably, comparing the word segmentation results of the base model's word segmenter on the domain corpus with the domain terminology database includes: The base model's word segmenter is invoked to segment the domain corpus sentence by sentence, outputting the offset correspondence between each sub-word and the original text's character interval. The character sequence of the term name is used to find the hit position in the sentence's character sequence by precise character-level matching. When adjacent terms have an inclusion relationship, the longest match priority principle is used to determine the hit result. Based on the offset correspondence, the start and end positions of the character interval are mapped to the sub-words covering the character interval. Sub-words that overlap with the term's character interval are included as a whole in the hit sub-word interval, thus obtaining the sub-word sequence of the hit term and the start and end positions of the sub-word interval.

[0007] Preferably, the calculation of segmentation fragmentation includes: pre-calling the word segmenter of the base model to traverse the general corpus, accumulating the occurrence count of each sub-word in the vocabulary, and storing it as a sub-word count table; determining the segmentation fragmentation as a weighted sum of a sub-word quantity term and a sub-word rarity term, wherein the sub-word quantity term is determined by the number of sub-words segmented from the hit term, and the sub-word rarity term is determined by the occurrence count of the sub-word in the sub-word count table; setting a preset threshold as the quantile value corresponding to the segmentation fragmentation distribution of all hit terms, and adding the condition that the occurrence count of the term in the domain corpus is not lower than a preset lower limit, and obtaining the target term after filtering.

[0008] Preferably, the step of encoding the definition text of the target term by the base model to generate semantic anchor points for the term includes: concatenating the name of the target term and the definition statement into definition text in the format of term name first and definition statement last, and truncating it according to a preset length limit, recording the word position range of the definition statement part in the definition text; inputting the definition text into the base model with frozen weights to perform a forward calculation, and taking the hidden state of the word position corresponding to the definition statement part in the last hidden layer; performing average pooling on the hidden state of the definition statement part according to word position to obtain the semantic anchor point vector of the target term.

[0009] Preferably, the extraction of the target term's sub-word representation in the domain corpus includes: extracting all hit sentences of the target term; deduplicating the hit sentences and grouping them into batches by word length in ascending order; padding the end of sentences within the same batch with placeholder words to reach the maximum length within the batch and generating a padding mask to mark the placeholder words; inputting the batched sentences into the weight-frozen base model to perform forward computation; extracting the hidden state vector corresponding to each sub-word at the hit position from the last hidden layer based on the sub-word interval of the target term in the hit sentences; and excluding the hidden state at the placeholder word position based on the padding mask.

[0010] Preferably, the aggregation to form a holistic representation of the term includes: weighting and summing the sub-word representations according to their occurrence positions to obtain a holistic representation for each occurrence, with the aggregation weight being the ratio of the number of characters in the sub-word to the sum of the number of characters in all sub-words of the target term; when the target term appears more than once in the domain corpus, averaging the holistic representations of each occurrence to obtain a holistic representation of the target term; and subtracting the semantic anchor point of the term from the holistic representation of the term to obtain a deviation vector, with the deviation direction being the normalized result of the deviation vector and the deviation magnitude being the L2 norm of the deviation vector.

[0011] Preferably, freezing the word segmenter and weights includes: keeping the word list, merging rules, sentence beginning tags and sentence ending tags of the base model's word segmenter unchanged, without adding new word list entries; and not registering gradients for the base model's input embedding matrix, attention sublayer parameters, feedforward sublayer parameters, root mean square normalization parameters and output projection matrix, with the values ​​remaining unchanged during fine-tuning.

[0012] Preferably, the construction of the correction mapping for the sub-word embedding of the target term includes: the correction mapping consists of a term matching mechanism and term-by-term correction parameters, configuring a correction direction vector and a correction intensity scalar for each target term, the dimension of the correction direction vector being consistent with the dimension of the input embedding; the term matching mechanism constructs a prefix tree using the sub-word identifier sequence of each target term as the key, with the end node of the key attached with the sequence number of the corresponding target term as a termination mark; during forward computation, the input lexical sequence is matched downwards along the prefix tree, and the continuous lexical segments that match the termination mark are determined as the hit span of the target term, and when nested matching exists, the longest matching is processed first.

[0013] Preferably, the correction mapping includes: the correction mapping is positioned after the input embedding layer and before the first decoding layer in the forward computation; the input lexical sequence obtains the sub-word static embedding vector by looking up the table in the input embedding matrix; after the term matching mechanism locates the hit span of the target term, a correction term is superimposed on the sub-word static embedding vector at each position within the hit span; the correction term is the product of the correction intensity scalar and the correction direction vector; the corrected embedding vector replaces the original static embedding vector and is sent to the decoding layer; the embedding vectors at the missing positions are not modified.

[0014] Preferably, the initialization of the correction direction and correction intensity of the correction mapping with the deviation direction and deviation magnitude respectively includes: setting the initial value of the correction direction vector to the deviation direction of the target term; setting the initial value of the correction intensity scalar to the product of the scaling factor and the deviation magnitude of the target term, wherein the scaling factor compresses the norm of the initial correction term to a set proportion of the norm of the sub-word static embedding vector; and registering the correction direction vector and correction intensity scalar as trainable parameters, which are continuously updated in joint fine-tuning.

[0015] Preferably, the joint fine-tuning of the correction mapping and low-rank adaptation parameters using domain corpus includes: a low-rank incremental branch is connected in parallel with the attention query projection matrix and value projection matrix in all decoding layers of the base model. The low-rank incremental branch is composed of a dimensionality reduction matrix and an dimensionality increase matrix concatenated. The dimensionality reduction matrix maps the input to the rank dimension, and the dimensionality increase matrix maps the rank dimension back to the input dimension. The output of the incremental branch is multiplied by a fixed scaling factor and then superimposed on the output of the main branch. The dimensionality reduction matrix is ​​initialized according to a Gaussian distribution with a mean of zero, and the dimensionality increase matrix is ​​initialized to all zeros.

[0016] Preferably, the term-level alignment loss includes: extracting the hidden state of the sub-word at the hit position from the last hidden layer based on the hit span output by the term matching mechanism, calculating the overall representation at the hit position with the same aggregation weight as when the term is aggregated into an overall representation, and averaging the overall representation when the same target term appears more than once in the current batch; the term-level alignment loss adopts the cosine distance between the overall representation at the hit position and the semantic anchor point of the term, and sums the values ​​of all target terms hit in the current batch according to the alignment weight; the alignment weight is obtained by normalizing the segmentation fragmentation degree of the target term and the average segmentation fragmentation degree of all target terms.

[0017] Preferably, the sentence-level language modeling loss includes: the sentence-level language modeling loss adopts the negative log-likelihood form of the autoregressive model, the forward calculation is performed according to the causal mask, and each position predicts the word at the current position based on the word before the current position; the joint fine-tuning loss is composed of the weighted sum of the sentence-level language modeling loss and the term-level alignment loss, and the term-level alignment loss is configured with a balance coefficient.

[0018] Preferably, the joint fine-tuning includes: using the AdamW optimizer, applying two sets of learning rates to the corrective mapping and the low-rank adaptation parameters, with the learning rate for the corrective mapping set higher than that for the low-rank adaptation parameters, and both sets of learning rates decaying linearly; setting a learning rate warm-up at the beginning of training, during which the learning rate linearly increases from zero to a set value; and performing global norm pruning on the norm of the concatenated gradients of all trainable parameters, scaling proportionally when the norm exceeds the pruning limit.

[0019] Preferably, the domain adaptation model includes: the corrected mapping parameters and the low-rank adaptation parameters are saved and together with the frozen base model to form the domain adaptation model; during deployment, the dimensionality reduction matrix and the dimensionality increase matrix of the low-rank incremental branch are multiplied by a fixed scaling factor and then accumulated to the corresponding attention query projection matrix and value projection matrix; the corrected mapping is loaded in a bypass form, and the hit position of the target term is located in the forward computation, and the word embeddings at the hit position are superimposed and corrected.

[0020] Secondly, this application also provides a domain data-driven large model adaptation system for implementing the domain data-driven large model adaptation method, including: a terminology screening module, a deviation measurement module, a correction construction module, and a joint fine-tuning module. The terminology filtering module is used to compare the word segmentation results of the base model's word segmenter on the domain corpus with the domain terminology database, count the number of sub-words segmented by the hit term and the number of times the sub-words appear in the general corpus, calculate the segmentation fragmentation degree, and filter target terms whose segmentation fragmentation degree exceeds a preset threshold. The deviation measurement module is used to encode the definition text of the target term by the base model, generate the semantic anchor point of the term, extract the word representation of the target term in the domain corpus, aggregate it into the overall representation of the term, and compare the overall representation of the term with the semantic anchor point of the term to obtain the deviation direction and deviation magnitude. The correction construction module is used to construct a correction mapping that acts on the word embedding of the target term under the condition that the word segmenter and weight of the base model are frozen, and to initialize the correction direction and correction intensity of the correction mapping with the deviation direction and deviation magnitude respectively. The joint fine-tuning module is used to jointly fine-tune the correction mapping and low-rank adaptation parameters using domain corpus. The loss of joint fine-tuning includes term-level alignment loss and sentence-level language modeling loss, which are weighted by the fragmentation degree and represent the overall term representation relative to the semantic anchor point of the term, respectively, to obtain the domain adaptation model.

[0021] The beneficial effects of this application are as follows: The technical solution of this application can repair the representation distortion caused by fragmentation of professional terms with a very low proportion of external parameters without changing the weights of the word segmenter and the base model. This enables the general large model to be adapted to different professional fields at low cost, significantly improving the accuracy of understanding domain terms and the quality of downstream text classification and summary generation. Moreover, the external parameters can be loaded or unloaded according to the domain, which facilitates the rapid switching and large-scale deployment of the same base model in multiple professional fields. Attached Figure Description

[0022] Figure 1 A flowchart of a domain data-driven large model adaptation method provided in this application; Figure 2 This is a schematic diagram illustrating the mapping and hit comparison between character ranges and sub-words in this application; Figure 3 This is a schematic diagram illustrating the fragmentation distribution and quantile threshold screening in this application; Figure 4 This is a schematic diagram of average pooling in the definition statement section of this application; Figure 5 This is a schematic diagram illustrating the deviation measurement of terminology used in this application; Figure 6 This is a schematic diagram of the prefix tree structure of the term matching mechanism in this application; Figure 7 This is a schematic diagram showing the superimposed location of the correction mapping in this application; Figure 8 This is a schematic diagram of the bypass parallel structure for low-rank adaptation in this application; Figure 9 This is a schematic diagram illustrating the relationship between the loss and trainable parameters in the joint fine-tuning of this application; Figure 10 This is a schematic diagram illustrating the application scenario of text processing in this application; Figure 11 This application provides a domain data-driven large model adaptation system architecture diagram. Detailed Implementation

[0023] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0024] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0025] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of this application. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single embodiment or an embodiment selectively excluded from other embodiments.

[0026] Example 1 Reference Figures 1 to 10 This is the first embodiment of the present application, providing a domain data-driven method for adapting large models. Figure 1 This is a flowchart of a domain data-driven large model adaptation method in this embodiment.

[0027] This embodiment provides a domain data-driven large model adaptation method. While keeping the base model's word segmenter and all original weights unchanged, it adapts the large language model to the target domain by constructing an embedding-side correction mapping and fine-tuning it in conjunction with low-rank adaptation parameters, thus obtaining a domain-adapted model.

[0028] In the pre-training phase, the word segmenter of a large language model constructs a vocabulary based on a general corpus, primarily covering common high-frequency words. When the base model is used in a specialized domain, domain-specific terms appear infrequently in the general corpus. The word segmenter often fails to include these terms as complete sub-words, resulting in fragmented sub-words when they are input into the base model. For example, a domain term might be segmented into four sub-words. The overall semantics of the term are diluted across these fragmented sub-words, leading to a bias in the base model's understanding of the term and consequently affecting the quality of downstream text classification and generation. Related techniques typically involve expanding the vocabulary and retraining the embedding for new terms, or using low-rank adaptation for domain-specific fine-tuning of the base model. The former modifies the word segmenter and breaks compatibility with existing inference engines, while the latter does not specifically address the cause of fragmented term segmentation. This embodiment addresses the representation distortion problem caused by fragmented term segmentation by sequentially executing steps S1 to S4 to repair the term representation without altering the weights of the word segmenter and the base model.

[0029] This embodiment uses a text processing task as an example to illustrate the implementation of each step. The domain corpus used for training comes from anonymized work order texts and service guide texts. During the anonymization process, names, contact numbers, and ID numbers are replaced with uniform placeholders. These placeholders are not included in subsequent statistics. The model output is used for work order request classification and work order summary generation. It should be noted that the method in this embodiment can be adapted to other professional fields after replacing the domain corpus and domain terminology database.

[0030] The base model employs an autoregressive large language model with a decoder structure. The selection criteria for the base model are: adaptation training and deployment must be completed on a localized training server; the model parameter size must match the GPU memory capacity of the training server; and the base model must have available weights to support parameter injection from the embedding and attention sides. For example, the base model has approximately 7 billion parameters and contains 32 decoding layers. Each decoding layer consists of a multi-head self-attention sub-layer with causal masks and a feedforward sub-layer concatenated. The hidden state dimension is 4096. Each decoding layer is configured with 32 attention heads, each with a 128-dimensional dimension. The intermediate dimension of the feedforward sub-layer is 11008. The activation function of the feedforward sub-layer is a SiLU function in the form of a gated linear unit. Layer normalization uses root mean square normalization, and the numerical smoothing term of the root mean square normalization is taken as... The positional encoding uses rotational positional encoding with a fundamental frequency of 10000. The base model supports a maximum context length of 4096 tokens. The word segmenter is constructed using byte-pair encoding, with a vocabulary of 32000 sub-words. The vocabulary includes two special tokens: sentence beginning markers and sentence ending markers. The input embedding matrix has a shape of 32000 rows and 4096 columns, containing approximately 131 million parameters. The language model's output projection matrix has a shape of 4096 rows and 32000 columns. The input embedding matrix and the output projection matrix do not share parameters. The number of decoding layers, hidden state dimension, attention head configuration, feedforward sub-layer dimension, normalization method, positional encoding method, context length, and vocabulary size are all inherent configurations of the selected base model. This embodiment does not modify the structural configuration of the base model.

[0031] The training server is equipped with multiple large-memory GPUs, and training is organized in data parallelism among these GPUs. Forward computation and backpropagation are performed using mixed precision in bfloat16 format, while the optimizer state is stored in 32-bit floating-point format. The model's forward computation, gradient updates, and parameter storage are implemented based on a deep learning training framework. Mixed precision is used because the dynamic range of bfloat16 format is close to that of 32-bit floating-point, which can reduce memory usage while maintaining the stability of forward values. Storing the optimizer's first and second moments in 32-bit floating-point format avoids rounding down small-step updates in low precision.

[0032] In step S1, the training server first reads the domain corpus and the domain terminology database. Each record in the domain terminology database contains a term name and a definition statement, with the definition statement taken from the definition entries in the domain specification document. For example, the preprocessed domain corpus contains approximately 320,000 sentences, and the domain terminology database contains approximately 15,000 term records. Data reading and text cleaning are performed according to conventional text processing procedures, including sentence segmentation by punctuation marks and deduplication of repeated sentences, which will not be elaborated in this embodiment.

[0033] After data reading is complete, the training server calls the base model's word segmenter to segment the domain corpus sentence by sentence. The word segmenter outputs a sequence of sub-words along with the offset correspondence between each sub-word and the character range in the original text. This offset correspondence records the start and end character positions of each sub-word in the original sentence. Byte-pair encoding merges consecutive characters into sub-words, and one sub-word corresponds to a segment of consecutive characters in the original sentence. Therefore, there is a definite mapping between the sub-word sequence and the character sequence.

[0034] Term match matching is performed at the character level with exact matching. The training server searches for matching positions in the character sequence of the sentence using the character sequence of the term name. When adjacent terms have an inclusion relationship, the longest match is prioritized to determine the match result. The reason for prioritizing the longest match is that the character sequence of a short term may be part of the character sequence of a long term. If the longest match is not taken, the long term will be split into short terms and residual characters, resulting in a mismatch between the match interval and the actual boundary of the term. After obtaining the character-level match interval, the training server maps the start and end character positions of the character interval to the sub-words covering the character interval according to the offset correspondence, thereby determining the sub-word sequence and the start and end positions of the sub-word interval into which the matched term is segmented. When there is a case where the term boundary falls inside a sub-word, the training server includes the entire sub-word that overlaps with the term's character interval into the matched sub-word interval to avoid half a sub-word participating in subsequent statistics. The training server writes the term name, the matched sentence number, and the start and end positions of the sub-word interval into the term index table. For example, about 9,600 terms out of approximately 15,000 terms have matches within the domain corpus.

[0035] Figure 2 This demonstrates the correspondence between the character sequences of sentences in the domain corpus and the word sequences of the word segmenter. Figure 2 The top row represents the character squares of the sentence, with a segment of characters marked in parentheses as the term character range; the bottom row represents the sub-word squares output by the word segmenter, with varying square widths reflecting that a sub-word covers a continuous segment of characters in the original sentence. The two dotted lines represent the offset correspondence, mapping the start and end positions of the term character range to the sub-words that cover that range, resulting in the hit sub-word ranges marked below. Figure 2 A term is segmented into multiple sub-words, and sub-words that overlap with the term's character range are included in the hit sub-word range as a whole, thus avoiding the inclusion of half a sub-word in subsequent statistics.

[0036] In the statistical phase, the training server records the number of sub-words segmented for each hit term and queries the occurrence count of each sub-word in the general corpus. To provide a benchmark for the count query, the training server pre-builds a sub-word count table on the general corpus. The general corpus consists of publicly available news texts, encyclopedia texts, and web page texts from the same source as the pre-training corpus of the base model. The general corpus needs to be large enough because the count statistics need to cover low-frequency sub-words within the corpus; if the corpus is too small, the counting noise of low-frequency sub-words is too high. For example, the general corpus contains approximately 2 billion lexical units. The training server calls the word segmenter of the base model to traverse the general corpus, accumulating the count for each sub-word encountered according to its identifier in the corpus. After the traversal is complete, the cumulative occurrence count of each sub-word in the corpus is obtained and stored as a sub-word count table in the form of a key-value table from sub-word identifier to occurrence count. The key of the sub-word count table is the integer identifier of the sub-word in the corpus, and the value is the corresponding cumulative occurrence count. Sub-words that do not appear in the general corpus have a value of 0. The construction of the sub-word count table is a one-time offline process. The construction result is domain-independent, and the same sub-word count table can be reused when adapting to different domains.

[0037] In this embodiment, the segmentation fragmentation degree is defined as the weighted sum of the number of sub-words and the rarity of sub-words. The formula for calculating the segmentation fragmentation degree can be: ;in, Indicates the degree of fragmentation of the hit terms. This indicates the number of subwords obtained by the word segmenter after the hit term is segmented. Indicates the ordinal number of the subword within the hit term. Indicates the first The number of times each subword appears in the general corpus This represents the total number of occurrences of all subwords in the general corpus. This represents the weighting coefficient between the term "subword quantity" and the term "subword rarity". This represents the natural logarithm. In the aforementioned formula, [the logarithm is missing here]. Adding 1 is to avoid the undefined logarithmic operation caused by subwords having a count of 0. (Subword count term) The term is set to 0 when the term is not segmented (i.e., the number of subwords is 1), and approaches 1 as the number of subwords increases. The more subwords are segmented, the larger the number of subwords. The subword rarity term is calculated by dividing the logarithm of the inverse frequency of each subword by 0. Normalized to the 0-1 range, the fewer times a subword appears in the general corpus, the larger its inverse frequency logarithm and the larger its rarity term. High-frequency subwords, whose frequency is close to the total frequency of all subwords, have a rarity term approaching 0. Both have been normalized to the 0-1 range, and the segmentation fragmentation also falls within the 0-1 range. The weighting coefficients... Used to balance the contribution of the number of segments and the rarity of subwords to the segmentation fragmentation degree. When the value is 0, the fragmentation degree is determined solely by the rarity of the subwords. When the value is 1, the fragmentation degree is determined solely by the number of sub-words. The method for determining the value is as follows: On the reserved validation corpus, the segmentation fragmentation degree is calculated with different weights, and the distortion of representation of highly segmented fragmented terms is observed. Weights with good distortion discrimination ability are selected. For example, Take 0.5.

[0038] To illustrate the calculation process of fragmentation degree, consider this example: a domain term is segmented into 4 sub-words by a word segmenter. The number of sub-words is 0.75; the average value of the rarity of the 4 sub-words is approximately 0.28. Taking a value of 0.5, the fragmentation degree of this domain term is approximately 0.51. In contrast, a general term that is fully included as a single sub-word has a sub-word count of 1 and a sub-word count item of 0. Furthermore, the single sub-word appears frequently in the general corpus and its rarity item is close to 0. The fragmentation degree of this general term is close to 0, which is clearly distinguishable from the fragmented domain term.

[0039] In the screening phase, the training server performs distribution statistics on the segmentation fragmentation of all hit terms, setting a preset threshold as the value corresponding to the 80th quantile of the segmentation fragmentation distribution. Setting the preset threshold using quantiles has the advantage that quantiles do not depend on the absolute dimension of segmentation fragmentation; the number of selected terms is automatically determined by the distribution, keeping the scale of subsequent correction parameters within budget while ensuring that the screening focuses on terms with segmentation fragmentation at the tail end of the distribution. For example, a preset threshold corresponding to the 80th quantile is 0.5, resulting in approximately 1900 terms after screening at 0.5. The training server adds an occurrence frequency condition during screening, requiring that the frequency of a term in the domain corpus not be lower than a preset lower limit. Setting a preset lower limit considers that when the frequency of a term is too low, the statistics calculated around the term's occurrence position are significantly affected by the randomness of the context of a single sentence, resulting in insufficient stability. For example, a preset lower limit of 3 times, after filtering with this condition, results in approximately 1600 target terms. The training server writes the name of the target term, the sub-word sequence, the fragmentation value, and all hit positions into the term index table and persists it for subsequent steps S2 to S4 to read.

[0040] Figure 3 The distribution of segmentation fragmentation for all hit terms is presented in histogram form, with the horizontal axis representing segmentation fragmentation and the vertical axis representing the number of terms. The main body of the distribution is concentrated in the low to medium value range, gradually thinning towards the high value range to form a tail. The vertical dashed line represents the preset threshold, determined according to the quantile of the segmentation fragmentation distribution; the solid square bars to the right of the threshold, marked in parentheses, represent the target terms, corresponding to terms whose segmentation fragmentation is at the tail of the distribution. Figure 3Specific numerical values ​​are not marked in the text. Thresholds are set in the form of quantiles so that the number of selected terms is automatically determined according to the distribution, thereby keeping the scale of subsequent correction parameters within the budget.

[0041] Once step S1 is completed, the source of the distortion is located to a limited number of target terms. Subsequent processing can be carried out around the records in the term index table, avoiding the computational overhead of modeling each of the approximately 15,000 terms. Furthermore, the fragmentation degree is composed of two objective statistics: the number of sub-words and the frequency of sub-words, and the screening results are reproducible.

[0042] After proceeding to step S2, the training server first generates semantic anchors for each target term. The definition text is organized as follows: the name of the target term and the definition statement are concatenated in the format "Term Name: Definition Statement". The definition text is truncated to a preset length limit, which is set in conjunction with the length of the definition statement: definition statements in the domain terminology library are mainly 1 to 2 sentences, and the length limit needs to cover the common length of definition statements and control inference overhead. For example, the length limit is set to 128 tokens. During concatenation, the training server records the token position range of the definition statement portion in the definition text, which is used to locate the definition statement portion from the hidden state later. The training server inputs the definition text into the base model in inference mode and performs a forward computation. All weights of the base model remain frozen, and the forward computation is performed using the bfloat16 format. In inference mode, gradients are not calculated, intermediate activation back buffers are not saved, and only the output of the last hidden layer is retained. After the forward computation is completed, the training server takes the hidden state of the corresponding word position in the definition statement part of the last hidden layer of the base model, and performs average pooling on the hidden state of the definition statement part according to the word position. That is, the hidden state of each word in the definition statement part is summed according to the dimension and then divided by the number of words in the definition statement part to obtain the semantic anchor vector of the target term.

[0043] Figure 4 It demonstrates how semantic anchors for terms are generated. Figure 4 The definition text on the left is composed of term names and definition statements concatenated in the format of name first, definition statement last. The weight-frozen pedestal model is input to perform a forward calculation. Figure 4 The right side shows the hidden state squares for each lexical position in the last hidden layer. The term names are marked as not participating in pooling, and the definition statements are indicated in parentheses. The hidden states in the definition statements are averaged by lexical position to obtain the semantic anchors of the terms. The pooling scope is limited to the definition statements and excludes the term names.

[0044] Pooling is limited to the definition statement portion and excludes the term name portion. This limitation includes: the term names contain fragmented sub-words that have been segmented; the representations of these fragmented sub-words themselves are distorted; and the semantic anchors should reflect the standard semantics carried by the definition statement. Excluding the term names portion prevents the semantic anchors from being contaminated by distorted representations. The semantic anchor vector has the same dimension as the hidden state, which is 4096. The training server organizes the semantic anchor vectors of approximately 1600 target terms into a 1600-row, 4096-column anchor matrix according to the order of the terms in the term index table, and caches it in 16-bit floating-point format. The row corresponds to the first The semantic anchor vectors for each target term. 16-bit floating-point storage is used because the semantic anchors are only read as the comparison benchmark and loss target; 16-bit precision meets the needs of cosine similarity and norm calculations, and storage usage is halved compared to 32-bit. For example, the anchor matrix occupies approximately 13MB of disk space.

[0045] After semantic anchors are generated, the training server extracts the sub-word representations of the target terms in the domain corpus. Based on the terminology index table, the training server retrieves all matching sentences for the target terms. After deduplication, the matching sentences are binned and batched in ascending order of word length. The advantage of binning is that sentences within the same batch have similar lengths, reducing computational waste from padding. Shorter sentences within the same batch are padded with placeholder words at the end to reach the maximum batch length. The training server simultaneously generates padding masks to mark the placeholder words; these placeholder words do not participate in subsequent hidden state retrieval. For example, the inference batch size is 64, and the maximum sentence length is 512 words. This upper limit matches the sentence length distribution after sentence segmentation: after sentence segmentation, the sentence length in the domain corpus is concentrated within 512 words, and truncation of excess portions has limited impact on the contextual integrity of the matching position. After forward computation, the training server retrieves the hidden state vector corresponding to each sub-word at the matching position from the last hidden layer based on the sub-word intervals recorded in the terminology index table. The hidden states at placeholder word positions are excluded based on the padding mask. The sub-word representation and semantic anchor point are taken from the same hidden layer because the deviation from the alignment requirement requires that the two compared quantities be in the same representation space. The last hidden layer carries the sentence-level semantic integration result. Taking the same layer can ensure that the alignment result reflects the semantic level difference rather than the inter-layer scale difference.

[0046] After obtaining the sub-word representations, the training server aggregates these representations into a total term representation based on their occurrence position. The total representation for a single occurrence can be obtained using a weighted summation method: ;in, This represents the overall representation vector of the target term at a single occurrence. Indicates the position of a single occurrence. The hidden state vector corresponding to each sub-word Indicates the first Aggregation weight of individual words. Aggregation weight Take the first The ratio of the number of characters in a sub-word to the sum of the number of characters in all sub-words of the target term, where the sum of the number of characters in each sub-word is the total number of characters in the term name, and the sum of all aggregated weights is 1. The weighting is based on the principle that sub-words with more characters carry more literal information about the term, and allocating weights according to the number of characters can make the overall representation biased towards sub-words with more information. When the target term is only segmented into a single sub-word, the aforementioned weighted summation formula degenerates into the hidden state of that single sub-word itself. When the target term appears more than once in the domain corpus, the overall representation across occurrences can be obtained by averaging: ;in, This represents the overall representation vector of the target term. This indicates the total number of times the target term appears in the domain corpus. This indicates that a serial number has appeared. Indicates the first The overall representation vector obtained from the previous weighted summation formula is used for each occurrence. Averaging multiple occurrences helps suppress random shifts introduced by single contexts; the selection criteria in step S1 have already ensured this. The mean estimate has basic stability if it is not lower than 3.

[0047] In the comparison phase, the training server calculates the deviation vector by subtracting the semantic anchor of a term from its overall representation. The deviation vector can be obtained by calculating the difference as follows: ;in, Represents the deviation vector. This represents the semantic anchor vector of the target term. The deviation direction is taken as the normalized result of the deviation vector, denoted as... , equal Divide by To avoid numerical instability during normalization when the L2 norm of the deviation vector approaches 0, the training server adds a very small positive lower bound to the denominator. The deviation magnitude can be taken as the L2 norm of the deviation vector, denoted as... The L2 norm is denoted as the square root of the sum of squares of the vector elements. The deviation direction indicates the direction required for the overall term representation to move towards the semantic anchor, while the deviation magnitude reflects the degree of distortion in the term representation. Since both the semantic anchor vector and the overall term representation vector are taken from the last hidden layer of the base model and have the same dimension of 4096, the deviation vector also has a dimension of 4096, consistent with the input embedding dimension. Correlation statistics were performed on the deviation magnitude and segmentation fragmentation degree of approximately 1600 target terms. For example, the Pearson correlation coefficient between deviation magnitude and segmentation fragmentation degree is approximately 0.6. The statistical results indicate that terms with severe segmentation fragmentation degree have severe representational distortion, confirming the rationality of using segmentation fragmentation degree to locate distorted terms. The training server writes the deviation direction and deviation magnitude of each target term into the term index table.

[0048] Figure 5 This demonstrates the principle of representing deviations from measurements within the hidden state representation space. Figure 5 The two black dots represent the semantic anchor point and the overall term representation, respectively. Both are taken from the same hidden layer of the base model and reside in the same representation space. The arrow pointing from the overall term representation to the semantic anchor point is the deviation vector. The arrow direction is labeled as the deviation direction, indicating the direction required for the overall term representation to move towards the semantic anchor point; the arrow length is labeled as the deviation magnitude, reflecting the degree of distortion in the term representation. Comparing the semantic anchor point and the overall term representation within the same representation space ensures that the comparison results are not affected by inconsistencies in the representation space of the external encoder and provides directional and scaled numerical basis for subsequent correction.

[0049] Step S2 uses the base model itself as a reference to complete the distortion measurement. The semantic anchor and the overall representation of the term are generated by the same hidden layer of the same base model. The comparison results are not affected by the inconsistency of the external encoder representation space. Furthermore, the deviation direction and deviation magnitude provide a directional and scaled numerical basis for subsequent correction.

[0050] Step S3 constructs the correction mapping under the condition that the tokenizer and weights of the base model are frozen. The specific meaning of freezing is: the tokenizer's vocabulary, merging rules, sentence beginning tags, and sentence ending tags remain unchanged, with no new vocabulary entries added; the input embedding matrix, attention sublayer parameters, feedforward sublayer parameters, root mean square normalization parameters, and output projection matrix of the base model do not register gradients, and their values ​​remain unchanged during training. The considerations for freezing are: not modifying the vocabulary maintains compatibility with existing inference engines and key-value caching mechanisms; not modifying the base model weights avoids the erosion of general capabilities by domain fine-tuning; and it allows the same base model to be equipped with external parameters from different domains.

[0051] The correction mapping consists of two parts: a term matching mechanism and term-by-term correction parameters, which will be referred to as correction parameters below. Regarding the term-by-term correction parameters, the training server configures one correction direction vector and one correction intensity scalar for each target term. The dimension of the correction direction vector is consistent with the input embedding dimension, which is 4096 dimensions. Both the correction direction vector and the correction intensity scalar are registered as trainable parameters and stored in 32-bit floating-point format during training. All correction direction vectors are organized into a 1600-row, 4096-column direction matrix. The row corresponds to the first The correction direction vector of the target term, the organization of all correction intensity scalars as 1600-dimensional intensity vectors, the intensity vector of the first... The component corresponds to the first The correction strength scalar for each target term. Regarding the terminology matching mechanism, the training server constructs a prefix tree using the sub-word identifier sequence of each target term as the key. Each node in the prefix tree stores the mapping from sub-word identifiers to child nodes, and the terminal node of the key is appended with the corresponding target term's sequence number as a termination marker. During forward computation, the input lexical sequence is matched downwards along the prefix tree from each position. Continuous lexical segments that match the termination marker are determined as the hit span of the target term. The cost of scanning the sequence once is proportional to the sequence length. When nested matching exists, the longest match is prioritized; that is, if multiple termination markers can be matched from the same starting position, the one with the longest span is selected. The matching rule corresponds to the character-level longest match in step S1, thus ensuring that the same term span is located during the training and inference phases.

[0052] Figure 6 The structure of the prefix tree used in the term matching mechanism is shown. The prefix tree uses the sequence of sub-word identifiers for each target term as the key. Each edge downward from the root node corresponds to a sub-word identifier, and the nodes store the mapping from sub-word identifiers to child nodes. Figure 6 The terminal nodes of the two paths are represented by double circles, with the corresponding target term's sequence number as a termination marker. During forward computation, the input lexical sequence is matched downwards along the prefix tree, and consecutive lexical segments that match the termination marker are determined as the hit span of the target term. Figure 6 The longer path on the right illustrates that when nested matches exist, the longest match is prioritized; that is, when multiple ending markers are matched from the same starting position, the one with the longest span is selected.

[0053] The corrected mapping operates after the input embedding layer and before the first decoding layer in the forward computation. The input word sequence is used to obtain the static word embedding vector by looking up the input embedding matrix, and the term matching mechanism locates the first... After determining the hit span of the target term, a correction term is superimposed on the static embedding vector of the sub-words at each position within the hit span. The corrected embedding vector can be obtained by the following superposition formula: ;in, Indicates the number of hits within the span. Word-corrected embedding vectors Indicates the first The static embedding vector of each subword, Indicates the sequence number of the target term. Indicates the first The correction direction vector of the target term. Indicates the first The correction intensity scalar for each target term. The correction term superimposed at each position within the hit span is the same, which is the product of the correction intensity scalar and the correction direction vector. The corrected embedding vector replaces the original static embedding vector and is fed into the decoding layer of the base model. The embedding vectors at missed positions remain unchanged. In practice, the correction superposition is implemented using sparse index addition. The training server only performs vector addition at hit positions and does not traverse all positions. The correction term is applied to the embedding side because the embedding side is the entry point for fragmented sub-words into the model. Compensating for term semantics at the entry point allows the compensation amount to propagate to all 32 decoding layers with forward computation, without requiring layer-by-layer intervention.

[0054] Figure 7 This demonstrates the role of the corrective mapping in the forward computation. Figure 7 The bottom middle section represents the input embedding layer, the top section represents the first decoding layer, and the middle row of squares represents the static embedding vectors of subwords obtained by looking up the input lexical sequence. After the term matching mechanism locates the hit span, it adds a correction term to the embedding vector at each position within the hit span. The correction term is determined by the correction direction vector and the correction intensity scalar; the embedding vectors at positions that are not hit are not modified. Figure 7 Above the hit position of the hit span, the correction term to be superimposed is drawn. The corrected embedding vector replaces the original static embedding vector and is sent to the decoding layer. The position of action is after the input embedding layer and before the first decoding layer. The compensation amount can be propagated to all decoding layers with forward computation.

[0055] In the initialization phase, the training server initializes the correction direction vector with the deviation direction obtained in step S2, and... The initial value is set to the first Deviation direction of target terminology ; Initialize the correction intensity scalar with the deviation amplitude, and The initial value is set to the scaling factor and the first Deviation of target terms The product of the products, with the scaling factor denoted as . Introducing a scaling factor This is because the deviation magnitude is measured in the hidden state space, and directly using it as the correction strength on the embedding side would cause excessive initial perturbation, causing the language modeling behavior at the training starting point to deviate from the original base model. The norm of the initial correction term is compressed to a set proportion of the norm of the word static embedding vector, allowing the correction to take effect gradually during joint fine-tuning. For example, Setting it to 0.1, the norm of the initial correction term is approximately one-tenth of the corresponding deviation magnitude. It should be noted that the input embedding dimension and hidden state dimension of the base model are consistent, and the deviation direction vector can be directly used as the initial value of the correction direction vector. Initialization only provides a priori starting point; the residual difference between the embedding space and the hidden state space is corrected by the joint fine-tuning in step S4. The correction direction vector and correction intensity scalar are continuously updated during training. Compared to random initialization, prior initialization ensures stronger correction for terms with heavy fragmentation and distortion at the start of training, shortening the number of training steps required for convergence.

[0056] From a parameter scale perspective, each target term has 4097 corrected parameters: a 4096-dimensional direction vector plus one intensity scalar. For example, approximately 1600 target terms correspond to approximately 6.56 million corrected parameters. Based on a base model with approximately 7 billion parameters, the corrected parameters represent about 0.09% of the total parameters. In terms of forward computation, the corrected mapping only adds one step of prefix tree matching and vector addition at the hit position, resulting in a limited increase in inference latency. The corrected mapping is saved as an independent parameter file and can be loaded or unloaded by domain. After unloading, the behavior of the base model remains completely consistent with that before adaptation.

[0057] Step S4 performs joint fine-tuning of the correction mapping and low-rank adaptation parameters using the domain corpus. The low-rank adaptation parameters are set as follows: a low-rank incremental branch is connected in parallel with the attention query projection matrix and value projection matrix of all 32 decoding layers in the base model. The low-rank incremental branch is constructed by concatenating a dimensionality reduction matrix and an dimensionality increase matrix. The dimensionality reduction matrix has a shape of 4096 rows and 8 columns, mapping the 4096-dimensional input to the rank dimension. The dimensionality increase matrix has a shape of 8 rows and 4096 columns, mapping the rank dimension back to 4096 dimensions. The output of the incremental branch is multiplied by a fixed scaling factor and then superimposed onto the main output. The rank value is determined by the following criteria: too small a rank results in insufficient fitting, while too large a rank increases parameter overhead and poses a risk of overfitting. For example, a rank of 8 is used. The fixed scaling factor is set according to the magnitude matching between the output of the incremental branch and the output of the main output; for example, a fixed scaling factor of 2 is used. The dimensionality reduction matrix is ​​initialized with a Gaussian distribution with a mean of 0 and a standard deviation of 0.02, with the standard deviation value being of the same order of magnitude as the initial weights of the base model. The dimensionality increase matrix is ​​initialized to all zeros. At the starting point of training, the output of the dimensionality increase matrix is ​​a zero vector, and the output of the incremental branch is zero. Joint fine-tuning starts from a state equivalent to the original base model. For example, the total number of parameters in the dimensionality reduction and dimensionality increase matrices of the query projection branch and the value projection branch within a single decoding layer is 2 x 4096 x 8, or 65536. The two branches have a total of 131072 parameters. The low-rank adaptation parameters of the 32 decoding layers total approximately 4.19 million, and together with approximately 6.56 million correction parameters, they total approximately 10.75 million, accounting for 0.15% of the approximately 7 billion parameter base model parameters.

[0058] Figure 8 This diagram illustrates a parallel bypass structure with low-rank adaptation parameters. The main path consists of the attention query projection matrix and the value projection matrix, with the main path parameters frozen. The bypass is a low-rank incremental branch, derived from the main path input. It is constructed by concatenating a dimensionality reduction matrix and an dimensionality increase matrix. The dimensionality reduction matrix maps the input to the rank dimension, and the dimensionality increase matrix maps the rank dimension back to the input dimension. The output of the incremental branch is scaled by a fixed factor and then superimposed onto the main path output at nodes marked as superposition. The diagram shows the connection relationships between the input, main path, bypass, and output, illustrating that the incremental branch is connected in parallel without altering the frozen parameters of the main path.

[0059] In terms of training data organization, the training server divides the domain corpus into a training set and a validation set. The validation set ratio balances two requirements: it needs to cover a sufficient number of term hit sentences to stabilize evaluation statistics, while not excessively crowding out the training data. For example, the validation set accounts for 5%, approximately 16,000 sentences. Training samples are concatenated and packaged sentence by sentence, with a maximum context length of 1024 units, determined based on the training GPU memory and the paragraph length of the domain text. Batch sampling uses a mixture of hit sentences and ordinary sentences. Hit sentences are those containing the target term, while ordinary sentences are those not containing the target term. The reason for controlling the mixing ratio is that the term-level alignment loss only generates gradients at the hit positions. If the proportion of hit sentences is too low, the alignment signal will be sparse; if the proportion is too high, the intra-batch text distribution will deviate from the natural distribution of the domain corpus. For example, the ratio of hit sentences to ordinary sentences is 1:3. Each GPU carries 4 samples per step, and the total training batch size for 8 GPUs is 32. The number of samples per GPU is determined based on the capacity of 24GB of GPU memory with a length of 1024 units.

[0060] Within each training step, the input sequence is first corrected by mapping to obtain the corrected embedding sequence, and then forward computation is completed by the decoding layer with a low-rank incremental branch. Each forward pass simultaneously produces two supervision signals. The first supervision signal is the sentence-level language modeling loss, which adopts the negative log-likelihood form of an autoregressive model. The formula for calculating the sentence-level language modeling loss is: ;in, This represents the sentence-level language modeling loss. This indicates the total number of terms involved in the loss calculation for the current batch. Indicates the position number of the lexical unit. Indicates the first The word element at each position, Indicates model first Predicting the first word under the condition of word 1 The conditional probability of each word is calculated by forward computation using a causal mask, with each position predicted only based on the word preceding it.

[0061] The second supervisory signal is the term-level alignment loss. Based on the hit span output by the term matching mechanism, the training server extracts the hidden state of the sub-word at the hit position from the last hidden layer, calculates the overall representation at the hit position using the same aggregation weight as in step S2, and averages the overall representations when the same target term appears more than once in the current batch, obtaining the... The overall representation of the target term under the combined effect of the corrected mapping and the low-rank increment is denoted as: It should be noted that, The term-level alignment loss is generated in the forward computation after the corrected mapping takes effect, and is distinct from the uncorrected global representation in step S2. The gradient of the term-level alignment loss is aggregated and backpropagated to the corrected direction vector, the corrected intensity scalar, and the low-rank adaptation parameters via the decoding layer. The term-level alignment loss can be expressed in cosine distance form, and its calculation formula is as follows: ;in, This represents the term-level alignment loss, summed by iterating through all target terms hit in the current batch. Indicates the first The overall representation vector after terminology correction for each target item. Indicates the first The semantic anchor vectors of the target terms are read from the anchor matrix cached in step S2 according to the term index. Cosine similarity represents the similarity between two vectors. Indicates the first Alignment weights for target terms. When cosine similarity is 1, the two vectors have the same direction and the corresponding loss term is 0. The cosine distance form constrains alignment only by direction, not by forcing the two vectors to have equal magnitudes. Alignment weights are obtained by normalizing the segmentation fragmentation. Alignment weights can be set to the [number]th [value]. The fragmentation degree of the target term Average fragmentation of all target terms The ratio is used. The reason for normalizing to the mean is that after normalization, the alignment weights fluctuate around 1, and fragmented terms achieve a higher-than-average alignment strength. At the same time, the overall magnitude of the weights is stable, and the loss coefficients do not need to be recalibrated as the term set changes. It should be noted that the weights of the base model are frozen throughout the process, the semantic anchors remain unchanged during training, the alignment target is fixed, and the term-level alignment loss will not experience target drift.

[0062] The joint loss is a weighted sum of the two losses, and the formula for calculating the joint loss is: ;in, Indicates joint loss, This represents the balance coefficient. The balance coefficient is set based on the following: balance coefficient Used to adjust the relative strength between term alignment and sentence-level language modeling. When the size is too large, the model excessively pursues the alignment of terminology representations with semantic anchors, which impairs the fluency of text generation. If the correction is too small, the correction strength is insufficient. On the validation set, the domain perplexity and term completion accuracy can be jointly selected. For example, Take 0.5.

[0063] For optimization, the AdamW optimizer was used during training, with a first-order moment decay coefficient of 0.9, a second-order moment decay coefficient of 0.999, and a numerical stability term of [missing value]. The weight decay is set to 0.01, a common configuration for fine-tuning deep networks, ensuring smooth parameter updates. Two sets of learning rates are used for the corrective mapping and low-rank adaptation parameters. The learning rate for the corrective mapping is set higher than that for the low-rank adaptation parameters. These two sets of learning rates are distinguished because the corrective parameters are fewer in number and have prior initialization, allowing them to withstand larger update steps to accelerate alignment. The low-rank adaptation parameters act on the attention backbone, and their update step size needs to be controlled to avoid sentence-level capability drift. For example, the learning rate for the corrective mapping is set to... The learning rate of the low-rank fitting parameter is taken as Both learning rates decay linearly. A learning rate warm-up is implemented at the beginning of training, where the learning rate linearly increases from 0 to a set value. This warm-up is necessary because the optimizer moment estimation is not yet stable in the early stages of training, and directly applying the full step size might disrupt the prior initialization of the correction parameters. For example, 200 warm-up steps are used. During training, global norm clipping is performed on the gradients. When the norm of the concatenated gradients of all trainable parameters exceeds the clipping upper limit, it is scaled proportionally. Setting the clipping upper limit takes into account the presence of colloquial noise sentences in the domain corpus, which may cause loss spikes in individual batches; norm clipping prevents parameter abrupt changes. For example, the clipping upper limit is set to 1.0. Three training epochs are used, with the number of epochs determined based on the convergence of the validation set metrics.

[0064] In the evaluation phase, the training server evaluates the domain perplexity and terminology completion accuracy on the validation set every half round. Domain perplexity is the exponent of the sentence-level language modeling loss on the validation set, reflecting the model's fit to the domain text. The terminology completion accuracy is evaluated as follows: sentences containing the target term are extracted from the validation set, with a maximum of two sentences extracted for each target term. This extraction limit is set to prevent high-frequency terms from dominating the statistics. The target term positions in the masked sentences are then filled in by the model using a greedy decoding method. A correct result is counted when the completed sentence precisely matches the original term string, and the correct percentage is statistically analyzed. Training is stopped early if the domain perplexity does not decrease after two consecutive evaluations. The criterion of two consecutive evaluations is used because a single fluctuation may stem from the randomness of batch sampling; convergence is only considered achieved after two consecutive evaluations without a decrease.

[0065] In joint fine-tuning, the two loss paths are coupled on a shared forward computation graph: the gradient of the sentence-level language modeling loss is backpropagated via the embedding vector at the hit position, constraining the correction parameters not to disrupt local language modeling; the gradient of the term-level alignment loss simultaneously reaches both the correction parameters and the low-rank adaptation parameters, guiding the low-rank incremental branch to adjust the attention behavior in conjunction with the corrected term representation. Since the correction parameters only act on the hit position and the low-rank adaptation parameters act on the attention projections at all positions, the two types of parameters obtain gradients according to their respective positions in the forward graph during the same backpropagation. Term-level representation correction and sentence-level semantic adaptation proceed simultaneously in the same gradient update, without requiring phased alternating training.

[0066] Figure 9 This paper demonstrates the relationship between the two-way loss and trainable parameters in joint fine-tuning. A single forward computation produces two supervision signals on a shared forward computation graph: sentence-level language modeling loss and term-level alignment loss. The term-level alignment loss is weighted by the fragmentation degree, and its alignment target is a fixed term semantic anchor point. Figure 9 The dashed line represents gradient backpropagation: the gradient of the sentence-level language modeling loss is backpropagated to the correction mapping parameters via the hit location, while the gradient of the term-level alignment loss reaches both the correction mapping parameters and the low-rank adaptation parameters simultaneously. Both the correction mapping parameters and the low-rank adaptation parameters are trainable parameters. The base model weights and semantic anchors remain unchanged during training, and term-level representation correction and sentence-level semantic adaptation are advanced simultaneously in the same backpropagation.

[0067] After training, the corrected mapping parameters and low-rank adaptation parameters, along with the term index table, are saved together and, together with the frozen base model, constitute the domain adaptation model. During deployment, the dimensionality reduction matrix and the dimensionality increase matrix of the low-rank incremental branch are multiplied by a fixed scaling factor and then directly added to the corresponding attention query projection matrix and value projection matrix weights. After merging, the inference path is consistent with the original base model, without increasing the computational load on the attention side. The corrected mapping is loaded as a bypass, and prefix tree matching and sparse addition are performed after the embedding layer. For example, the corrected mapping parameters, exported as 16-bit floating-point numbers, occupy approximately 13MB, and the low-rank adaptation parameters, exported as 16-bit floating-point numbers, occupy approximately 8.4MB, for a total of approximately 21.4MB of external files. When switching domains, only the external parameter files are replaced; the base model does not need to be redistributed.

[0068] For example, on the test set for the current domain text processing task, the domain perplexity of the adapted model decreased from 16.8 to 10.5, and the term completion accuracy increased from 38% to 61%. The perplexity increase on the general test set was less than 0.3, indicating that term-level correction and sentence-level adaptation did not come at the expense of general applicability. When the adapted model was used for classifying work order requests, the classification accuracy improved compared to before adaptation.

[0069] Figure 10 This embodiment demonstrates the application scenarios of its technical solution. Figure 10 The input on the left is the work order text and service guide text, which are anonymized before being entered into the domain adaptation model. The domain adaptation model consists of a frozen base model and external parameters, namely the correction mapping and low-rank adaptation parameters. Below, the current domain external parameters and other domain external parameters are given. These can be loaded or unloaded by domain, allowing the same base model to switch between multiple professional domains. Figure 10 The output on the right side of the page contains two types of downstream results: work order request classification and work order summary generation. Figure 10The connecting lines in the middle represent the relationship between text input, model processing and downstream output, as well as the loading and unloading characteristics of external parameters.

[0070] By combining steps S1 to S4, this embodiment forms a complete processing chain of distortion localization, distortion measurement, correction construction, and correction consolidation: the segmentation fragmentation degree is used as a screening criterion in step S1 and reused as alignment weight in step S4; the deviation direction and deviation magnitude are measured in step S2 and converted into prior initial values ​​of correction parameters in step S3; and the intermediate quantities of each step are interconnected. The entire adaptation process does not modify the weights of the word segmenter and the base model, and the proportion of newly added parameters is approximately 0.15%, making training and deployment costs controllable; while the overall representation of terms converges to the semantic anchor point, the sentence-level language modeling capability is simultaneously adapted to the domain corpus, thus balancing the fidelity of term representation with the quality of text processing.

[0071] Example 2 Reference Figure 11 This is the second embodiment of the present application, which provides a domain data-driven large model adaptation system.

[0072] The system includes: a terminology screening module, a deviation measurement module, a correction construction module, and a joint fine-tuning module.

[0073] The terminology filtering module is used to compare the word segmentation results of the base model's word segmenter on the domain corpus with the domain terminology database, count the number of sub-words segmented by the hit term and the number of times the sub-words appear in the general corpus, calculate the segmentation fragmentation degree, and filter target terms whose segmentation fragmentation degree exceeds a preset threshold.

[0074] The deviation measurement module is used to encode the definition text of the target term by the base model, generate the semantic anchor point of the term, extract the sub-word representation of the target term in the domain corpus, aggregate it into the overall representation of the term, and compare the overall representation of the term with the semantic anchor point of the term to obtain the deviation direction and deviation magnitude.

[0075] The correction construction module is used to construct a correction mapping that acts on the word embedding of the target term under the condition that the word segmenter and weight of the base model are frozen, and to initialize the correction direction and correction intensity of the correction mapping with the deviation direction and deviation magnitude, respectively.

[0076] The joint fine-tuning module is used to jointly fine-tune the correction mapping and low-rank adaptation parameters using domain corpus. The loss of joint fine-tuning includes term-level alignment loss and sentence-level language modeling loss, which are weighted by the fragmentation degree and represent the overall term representation relative to the semantic anchor point of the term, respectively, to obtain the domain adaptation model.

[0077] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0078] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments under the guidance of this application without departing from the spirit and scope of protection of the claims. All of these variations are within the protection scope of this application.

Claims

1. A domain data-driven method for adapting large models, characterized in that, include: The word segmentation results of the base model's word segmenter on the domain corpus are compared with the domain terminology database. The number of sub-words segmented by the hit terms and the number of times the sub-words appear in the general corpus are counted. The segmentation fragmentation is calculated, and target terms with segmentation fragmentation exceeding the preset threshold are selected. The definition text of the target term is encoded by the base model to generate semantic anchors for the term; Extract the word representations of the target term from the domain corpus, aggregate them into a term-wide representation, and compare the term-wide representation with the term semantic anchors to obtain the deviation direction and deviation magnitude. With the word segmenter and weights of the base model frozen, a correction mapping for the word embedding of the target term is constructed, and the correction direction and correction intensity of the correction mapping are initialized with the deviation direction and deviation magnitude, respectively. The domain-specific corpus is used to jointly fine-tune the correction mapping and low-rank adaptation parameters. The joint fine-tuning loss includes term-level alignment loss weighted by segmentation fragmentation degree, term-level overall representation relative to term semantic anchors, and sentence-level language modeling loss, thus obtaining the domain adaptation model.

2. The domain data-driven large model adaptation method according to claim 1, characterized in that, The step of comparing the word segmentation results of the base model's word segmenter on the domain corpus with the domain terminology database includes: The base model's word segmenter is invoked to segment the domain corpus sentence by sentence, and the offset correspondence between each subword and the original text character range is output. The system searches for the hit position in the character sequence of the sentence by using the character sequence of the term name for a precise character-level match. When adjacent terms have an inclusion relationship, the hit result is determined by the longest match priority principle. Based on the offset correspondence, the start and end positions of the character interval are mapped to the sub-words covering the character interval. Sub-words that overlap with the term character interval are included in the hit sub-word interval, thus obtaining the sub-word sequence of the hit term and the start and end positions of the sub-word interval.

3. The domain data-driven large model adaptation method according to claim 1, characterized in that, The calculation of the fragmentation degree includes: The base model's word segmenter is called in advance to traverse the general corpus, accumulate the occurrence count of each sub-word in the vocabulary, and store it as a sub-word count table; The segmentation fragmentation is determined as a weighted sum of the sub-word count and the sub-word rarity. The sub-word count is determined by the number of sub-words that are segmented from the hit term, and the sub-word rarity is determined by the number of times the sub-word appears in the sub-word count table. The preset threshold is set to the quantile value corresponding to the fragmentation distribution of all hit terms, and the condition that the number of times the term appears in the domain corpus is not less than the preset lower limit is added. After filtering, the target terms are obtained.

4. The domain data-driven large model adaptation method according to claim 1, characterized in that, The definition text of the target term encoded by the base model, generating term semantic anchors, includes: The name of the target term and the definition statement are concatenated into the definition text in the format of term name first and definition statement second, and truncated according to the preset length limit. The word position range of the definition statement part in the definition text is recorded. Perform a forward computation on the base model with the weights of the defined text input frozen, and take the hidden state of the corresponding word position in the definition statement part in the last hidden layer; The hidden states of the definition statement are averaged by word position to obtain the semantic anchor vector of the target term.

5. The domain data-driven large model adaptation method according to claim 1, characterized in that, The extraction of sub-word representations of the target term in the domain corpus includes: Extract all sentences that match the target term, remove duplicates from the sentences, and group them into batches in ascending order of word length. For sentences within the same batch, pad the end with placeholder words to make it the maximum length within the batch, and generate a padding mask that marks the placeholder words. The batched sentences are input into the weight-frozen base model for forward computation. Based on the sub-word intervals of the target term in the hit sentence, the hidden state vector corresponding to each sub-word at the hit position is extracted from the last hidden layer. The hidden state at the placeholder word position is excluded based on the padding mask.

6. The domain data-driven large model adaptation method according to claim 1, characterized in that, The aggregation as a holistic representation of the terminology includes: The sub-word representations are weighted and summed according to their occurrence position to obtain the overall representation of a single occurrence. The aggregation weight is the ratio of the number of characters in the sub-word to the sum of the number of characters in all sub-words of the target term. When a target term appears more than once in the domain corpus, the overall representation of each occurrence is averaged to obtain the overall representation of the target term. The deviation vector is obtained by subtracting the semantic anchor of the term from the overall representation of the term. The deviation direction is taken as the normalized result of the deviation vector, and the deviation magnitude is taken as the L2 norm of the deviation vector.

7. The domain data-driven large model adaptation method according to claim 1, characterized in that, The freezing of both the word segmenter and weights includes: The vocabulary, merging rules, sentence beginning and sentence ending tags of the base model's word segmenter remain unchanged, without adding new vocabulary entries; The input embedding matrix, attention sublayer parameters, feedforward sublayer parameters, root mean square normalization parameters, and output projection matrix of the base model do not register gradients, and their values ​​remain unchanged during fine-tuning.

8. The domain data-driven large model adaptation method according to claim 1, characterized in that, The construction of the corrective mapping that operates on the sub-word embeddings of the target term includes: The correction mapping consists of a term matching mechanism and term-by-term correction parameters. Each target term is configured with a correction direction vector and a correction intensity scalar. The dimension of the correction direction vector is consistent with the input embedding dimension. The term matching mechanism constructs a prefix tree using the sequence of sub-word identifiers for each target term as the key, with the terminal node of the key appended with the corresponding target term's index as a termination marker; During forward computation, the input lexical sequence is matched downwards along the prefix tree. Continuous lexical segments that match the termination marker are determined as the hit span of the target term. When nested matching exists, the longest match is processed first.

9. The domain data-driven large model adaptation method according to claim 8, characterized in that, The corrective mapping includes: The correction mapping operates in the forward computation after the input embedding layer and before the first decoding layer. The input word sequence is used to obtain the static embedding vector of the sub-word by looking up the table of the input embedding matrix. After the term matching mechanism locates the hit span of the target term, a correction term is superimposed on the static embedding vector of the sub-word at each position within the hit span. The correction term is the product of the correction intensity scalar and the correction direction vector. The corrected embedding vector replaces the original static embedding vector and is sent to the decoding layer; the embedding vectors at the missed positions are left unchanged.

10. The domain data-driven large model adaptation method according to claim 1, characterized in that, The method of initializing the correction mapping with the deviation direction and deviation magnitude respectively includes: Set the initial value of the correction direction vector to the deviation direction of the target term; The initial value of the correction intensity scalar is set to the product of the scaling factor and the deviation magnitude of the target term. The scaling factor compresses the norm of the initial correction term to a set proportion of the norm of the sub-word static embedding vector. The correction direction vector and correction intensity scalar are registered as trainable parameters and continuously updated during joint fine-tuning.

11. The domain data-driven large model adaptation method according to claim 1, characterized in that, The joint fine-tuning and correction mapping and low-rank adaptation parameters based on domain corpus include: In the base model, the attention query projection matrix and the value projection matrix of all decoding layers are connected in parallel with low-rank incremental branches. The low-rank incremental branches are composed of dimensionality reduction matrix and dimensionality increase matrix concatenated. The dimensionality reduction matrix maps the input to the rank dimension, and the dimensionality increase matrix maps the rank dimension back to the input dimension. The output of the incremental branch is multiplied by a fixed scaling factor and then superimposed onto the main output; The dimension-reducing matrix is ​​initialized with a Gaussian distribution with a mean of zero, while the dimension-increasing matrix is ​​initialized with all zeros.

12. The domain data-driven large model adaptation method according to claim 1, characterized in that, The term-level alignment loss includes: Based on the hit span output by the term matching mechanism, the hidden state of the sub-word at the hit position is extracted from the last hidden layer. The overall representation at the hit position is calculated using the same aggregation weight as when the term is aggregated into an overall representation. When the same target term appears more than once in the current batch, the overall representation is averaged. The term-level alignment loss is expressed as the cosine distance between the overall representation of the hit point and the semantic anchor of the term, and is weighted and summed for all target terms hit in the current batch according to the alignment weight. The alignment weight is obtained by normalizing the segmentation fragmentation of the target term with the average segmentation fragmentation of all target terms.

13. The domain data-driven large model adaptation method according to claim 1, characterized in that, The sentence-level language modeling loss includes: The sentence-level language modeling loss adopts the negative log-likelihood form of the autoregressive model. The forward computation is performed according to the causal mask, and each position predicts the word at the current position based on the word before the current position. The loss of joint fine-tuning consists of a weighted sum of sentence-level language modeling loss and term-level alignment loss, with a balancing coefficient configured before the term-level alignment loss.

14. The domain data-driven large model adaptation method according to claim 1, characterized in that, The joint fine-tuning includes: The AdamW optimizer is used, with two sets of learning rates applied to the corrective mapping and the low-rank fitting parameters. The learning rate for the corrective mapping is set higher than that for the low-rank fitting parameters, and both sets of learning rates decay linearly. At the beginning of training, a learning rate warm-up is set, during which the learning rate linearly increases from zero to the set value. Global norm clipping is performed on the norm of the concatenated gradients of all trainable parameters. If the norm exceeds the clipping limit, it is scaled proportionally.

15. The domain data-driven large model adaptation method according to claim 1, characterized in that, The domain adaptation model includes: After the corrected mapping parameters and low-rank adaptation parameters are saved, they together with the frozen base model to form the domain adaptation model; During deployment, the reduced-dimensionality matrix and the increased-dimensionality matrix of the low-rank incremental branch are multiplied by a fixed scaling factor and then accumulated into the corresponding attention query projection matrix and value projection matrix. The correction mapping is loaded in a bypass manner, locating the hit position of the target term in the forward computation, and superimposing correction on the word embeddings at the hit position.

16. A domain data-driven large model adaptation system, used to implement the domain data-driven large model adaptation method according to any one of claims 1 to 15, characterized in that, include: The module includes a terminology screening module, a deviation measurement module, a correction construction module, and a joint fine-tuning module. The terminology filtering module is used to compare the word segmentation results of the base model's word segmenter on the domain corpus with the domain terminology database, count the number of sub-words segmented by the hit term and the number of times the sub-words appear in the general corpus, calculate the segmentation fragmentation degree, and filter target terms whose segmentation fragmentation degree exceeds a preset threshold. The deviation measurement module is used to encode the definition text of the target term by the base model, generate the semantic anchor point of the term, extract the word representation of the target term in the domain corpus, aggregate it into the overall representation of the term, and compare the overall representation of the term with the semantic anchor point of the term to obtain the deviation direction and deviation magnitude. The correction construction module is used to construct a correction mapping that acts on the word embedding of the target term under the condition that the word segmenter and weight of the base model are frozen, and to initialize the correction direction and correction intensity of the correction mapping with the deviation direction and deviation magnitude respectively. The joint fine-tuning module is used to jointly fine-tune the correction mapping and low-rank adaptation parameters using domain corpus. The loss of joint fine-tuning includes term-level alignment loss and sentence-level language modeling loss, which are weighted by the fragmentation degree and represent the overall term representation relative to the semantic anchor point of the term, respectively, to obtain the domain adaptation model.

Images