Hybrid supervised language model distillation method and device based on bridge domain-park domain decomposition

Abstract

The application provides a mixed supervision language model distillation method based on bridge domain-park domain decomposition, comprising the following steps: constructing a prefix state sequence, and saving context and task identification. The state information is used for subsequent partition discrimination and error statistics. The probability distribution, hard label, confidence and entropy statistics of the teacher model are obtained for each state. The state risk is calculated, and the bridge domain and the park domain are divided to form a partition label. The partition label can be reused for training weighting and slice analysis. Soft supervision and hard supervision are calculated at the state level, and are dynamically weighted. The joint optimization process keeps the single model parameter space solution, and reduces the process complexity. The final model is selected according to the accuracy, stability, time delay and cost, and an auditable evaluation report is formed. In the evaluation stage, unified benchmarks and fixed random seeds are recommended to improve comparability.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and particularly to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for distilling a hybrid supervised language model based on bridge domain-circle domain decomposition. Background Technology

[0002] Large language models excel in reasoning and coding tasks, but their training and deployment costs continue to rise. Applications sensitive to latency, energy consumption, and computing power urgently require robust migration of teacher model capabilities to lightweight student models. This need is particularly prominent in government and enterprise private deployments, edge terminals, and real-time service scenarios.

[0003] Existing distillation paradigms mainly include hard-label supervision and soft-label supervision. Hard-label supervision offers robust convergence but has limited information density; soft-label supervision provides ample information but is prone to introducing high-risk biases in critical states. A single paradigm cannot simultaneously guarantee stability and distribution fidelity. Therefore, the design of distillation objectives must achieve a structured balance between information sufficiency and robust convergence.

[0004] Measuring distillation quality solely by the goodness of fit during the training phase is insufficient. The state distribution of the student model during the inference phase is determined by its own historical output, and a systematic bias exists between the training and inference distributions, leading to the accumulation of exposure bias. This conclusion suggests that distillation optimization should not only focus on single-step fitting but should also pay attention to sequence-level error propagation.

[0005] To explain this phenomenon, this invention proposes for the first time a bridge-circle decomposition theory: dividing the generated states into high-risk bridge domains and low-risk circular domains. Bridge domains emphasize precise selection and are best suited for hard supervision to suppress cascading errors; circular domains emphasize semantic flexibility and are best suited for soft supervision to maintain the distribution structure. Therefore, state partitioning is a crucial link connecting theoretical analysis and engineering strategies.

[0006] Therefore, a unified, engineerable mechanism is needed to dynamically integrate hard and soft supervision at the state level, and improve inference quality and deployment efficiency without significantly increasing the training burden. This lays the methodological foundation for building a unified and scalable distillation process. Summary of the Invention

[0007] This invention relates to a knowledge distillation method for autoregressive language models, and more particularly to a hardware-soft supervised collaborative training framework based on bridge domain-circle domain decomposition. This framework is designed for high-quality generation tasks under limited computing power, while balancing training reproducibility and deployment maintainability.

[0008] This invention aims to address the following key technical issues. First, to address the common mismatch phenomenon in autoregressive distillation—"good single-step distribution fitting but decreased performance in multi-step generation"—a risk modeling mechanism that can directly act on the state level is established, enabling training constraints to cover the sequence-level error propagation process. Second, to address the structural deficiencies of pure hard supervision and pure soft supervision, a hybrid supervision framework that adapts to state risk is constructed. This framework strengthens precise constraints in critical states and maintains distribution elasticity in non-critical states, thereby avoiding excessive contraction or over-diffusion. Third, to address the distribution heterogeneity and threshold sensitivity issues in multi-task joint distillation, a calibrable, hierarchical, and reproducible parameter mechanism is provided, enabling the same process to stably adapt to inference, mathematical, and coding tasks. Fourth, to address the constraints of training efficiency and deployment costs on the engineering side, an offline supervision caching and parallel supervision acquisition strategy are adopted to reduce repeated teacher calls and ensure process feasibility and iterative efficiency. This invention also emphasizes process verifiability and parameter interpretability: auditable metrics are retained for each training stage to ensure that the sources of model improvement are traceable, reproducible, and comparable. This invention balances theoretical interpretability with engineering feasibility, and can be directly embedded into existing distillation training platforms. At the implementation level, this invention emphasizes closed-loop management of indicators, ensuring that bias attribution and strategy backtracking can be performed in each training round. Simultaneously, through unified logs and evaluation criteria, this invention can continuously monitor strategy effectiveness and support iterative optimization.

[0009] To address the shortcomings of existing technologies, such as Figure 1 As shown, this invention proposes a hybrid supervised language model distillation method based on bridge domain-circle domain decomposition, which includes:

[0010] The initial step involves acquiring a natural language question-and-answer dataset, which includes multiple question-and-answer samples. Each sample contains a question and its corresponding standard answer. Following an autoregressive process, the standard answer is expanded into a prefix state sequence S = [s1, s2, ..., s...] token by token. T ], where T is the sequence length of the standard answer;

[0011] The prediction step involves processing each state s in the prefix state sequence S. t The teacher model and student model are called to calculate their probability distributions and hard labels a*, respectively, to obtain the teacher distribution, teacher labels and student distributions;

[0012] The loss step involves dividing all prefix state sequences S into small, medium, and large buckets according to their length, and sampling from only one bucket in each iteration; a soft loss function is constructed based on the approximation of the student distribution to the teacher distribution, and a hard loss function is constructed based on the negative log-likelihood of the student model to the hard label a*.

[0013] The training steps involve using the difference between the probability distributions of the teacher model and the student model in the same state as the word-level local risk sensitivity. If the local risk sensitivity is greater than a preset value, it's considered a bridge region. The hard loss function is multiplied by weight A and then summed with the soft loss function to form a hybrid loss function. Otherwise, it's considered a circle region. The hard loss function is multiplied by weight B and then summed with the soft loss function to form a hybrid loss function. Weight A is greater than weight B. The hybrid loss function is used to calculate the gradient through backpropagation and update the student model. The prediction, loss, and training steps are iterated repeatedly until the hybrid loss function converges. The current student model is then saved as the final model.

[0014] The question-and-answer process involves inputting the question text into the final model to obtain the answer to the question text.

[0015] The aforementioned hybrid supervised language model distillation method based on bridge domain-circle domain decomposition includes the following training steps:

[0016] Calculate the local risk sensitivity :

[0017]

[0018] Let κ(a|s) represent the candidate lexical units in the vocabulary V, and let κ(a|s) represent the local risk sensitivity resulting from choosing lexical unit a in state s.

[0019] Based on risk aggregation Delineating bridge domains With the Garden Area ;

[0020] ;

[0021] τ is a preset risk threshold, with a value ranging from 0.1 to 0.9;

[0022] The hybrid loss function :

[0023] ;

[0024] ;

[0025] θ represents the student model parameters, and λ(s) represents the soft supervision weights. It is a soft loss function. It is a hard loss function; Predicting words for the teacher model in state s The probability, This represents the maximum probability among all candidate lexical units in the teacher model.

[0026] The aforementioned hybrid supervised language model distillation method based on bridge domain-circle domain decomposition includes the following training steps:

[0027] Calculate the local risk sensitivity ;

[0028] Based on risk aggregation Delineating bridge domains With the Garden Area ;

[0029]

[0030] The hybrid loss function :

[0031] ;

[0032] ;

[0033] ;

[0034] .

[0035] In the formula The prediction entropy of the teacher model in state s. For vocabulary size, The maximum possible entropy value; This represents the corrected hard loss function; α is a positive correction strength coefficient. Let be the log-probability interval of the student model at state s and hard label a*. (·|s) represents the logit output value of the student model for each word in state s. This indicates the new state formed by appending the hard label a* to state s.

[0036] like Figure 2 As shown, this invention also proposes a hybrid supervised language model distillation device based on bridge domain-circle domain decomposition, comprising:

[0037] The initial module acquires a natural language question-and-answer dataset, which includes multiple question-and-answer samples. Each sample contains a question and its corresponding standard answer. Following an autoregressive order, the standard answer is expanded into a prefix state sequence S = [s1, s2, ..., s...]. T ], where T is the sequence length of the standard answer;

[0038] The prediction module, for each state s in the prefix state sequence S t The teacher model and student model are called to calculate their probability distributions and hard labels a*, respectively, to obtain the teacher distribution, teacher labels and student distributions;

[0039] The loss module divides all prefix state sequences S into small, medium, and large buckets according to their length, and samples from only one bucket in each iteration; it constructs a soft loss function based on how closely the student distribution approximates the teacher distribution, and a hard loss function based on the negative log-likelihood of the student model with respect to the hard label a*.

[0040] The training module uses the difference between the probability distributions of the teacher model and the student model in the same state as the word-level local risk sensitivity. It determines whether the local risk sensitivity is greater than a preset value. If it is, it's a bridge region; the hard loss function is multiplied by weight A and then summed with the soft loss function to form a hybrid loss function. Otherwise, it's a circle region; the hard loss function is multiplied by weight B and then summed with the soft loss function to form a hybrid loss function. Weight A is greater than weight B. The hybrid loss function is used to calculate the gradient through backpropagation and update the student model. The prediction module, loss module, and training module are iteratively executed repeatedly until the hybrid loss function converges, and the current student model is saved as the final model.

[0041] The question-and-answer module takes the question text as input into the final model and outputs the answer to that question text.

[0042] The aforementioned hybrid supervised language model distillation device based on bridge-circle domain decomposition, wherein the training module includes:

[0043] Calculate the local risk sensitivity :

[0044]

[0045] Let κ(a|s) represent the candidate lexical units in the vocabulary V, and let κ(a|s) represent the local risk sensitivity resulting from choosing lexical unit a in state s.

[0046] Based on risk aggregation Delineating bridge domains With the Garden Area ;

[0047] ;

[0048] τ is a preset risk threshold, with a value ranging from 0.1 to 0.9;

[0049] The hybrid loss function :

[0050] ;

[0051] ;

[0052] θ represents the student model parameters, and λ(s) represents the soft supervision weights. It is a soft loss function. It is a hard loss function; Predicting words for the teacher model in state s The probability, This represents the maximum probability among all candidate lexical units in the teacher model.

[0053] The aforementioned hybrid supervised language model distillation device based on bridge-circle domain decomposition, wherein the training module includes:

[0054] Calculate the local risk sensitivity ;

[0055] Based on risk aggregation Delineating bridge domains With the Garden Area ;

[0056]

[0057] The hybrid loss function :

[0058] ;

[0059] ;

[0060] ;

[0061] .

[0062] In the formula The prediction entropy of the teacher model in state s. For vocabulary size, The maximum possible entropy value; This represents the corrected hard loss function; α is a positive correction strength coefficient. Let be the log-probability interval of the student model at state s and hard label a*. (·|s) represents the logit output value of the student model for each word in state s. This indicates the new state formed by appending the hard label a* to state s.

[0063] The present invention also proposes a client for implementing any of the hybrid supervised language model distillation devices based on bridge domain-circle domain decomposition.

[0064] The present invention also proposes an electronic device, including the aforementioned distillation apparatus for a hybrid supervised language model based on bridge-circle decomposition, which may be connected to an information display device for displaying the answer result with user-set display parameters, attributes, or through an artificial intelligence model.

[0065] The present invention also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the described hybrid supervised language model distillation method based on bridge domain-circle domain decomposition.

[0066] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, it implements the steps of the hybrid supervised language model distillation method based on bridge domain-circle domain decomposition.

[0067] As can be seen from the above solutions, the advantages of the present invention are:

[0068] Compared to state-of-the-art distillation methods, this invention demonstrates stable and consistent gains across multiple model families and multi-task benchmarks. In addition to accuracy gains, it outperforms traditional online policy distillation methods in training efficiency while maintaining the reproducibility and maintainability of the offline training process. In practical deployments, this invention can be sequentially combined with quantization, pruning, and low-rank adaptation techniques to first perform capability transfer followed by structure compression, maintaining high task availability under limited computing power. This process exhibits good reproducibility and maintainability, making it suitable for continuous iteration. Under the same resource constraints, this method combines stable gains with high process efficiency. This characteristic makes it more valuable for long-term evolution projects. Under the same resource conditions, this method balances performance improvement with process robustness, good adaptability, and clear engineering value. Attached Figure Description

[0069] Figure 1 This is a flowchart of the method of the present invention;

[0070] Figure 2 This is a block diagram of the device of the present invention;

[0071] Figure 3 This is a schematic diagram of the structure of the first electronic device of the present invention;

[0072] Figure 4 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0073] Figure 5 This is a schematic diagram of the structure of the second electronic device of the present invention.

[0074] Figure label:

[0075] A - First electronic device;

[0076] B-Distillation apparatus for a hybrid supervised language model based on bridge-circle decomposition;

[0077] C-Data acquisition equipment;

[0078] D-Information display device;

[0079] 1000 - Second electronic device;

[0080] Ⅰ-Computational Unit;

[0081] II-ROM;

[0082] III-RAM;

[0083] N-bus;

[0084] V-Interface;

[0085] VI - Input Unit;

[0086] VII - Output Unit;

[0087] VIII - Storage medium;

[0088] IX - Communication Unit. Detailed Implementation

[0089] It should be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0090] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0091] The processor described in this invention is the control center of an electronic device. It can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of this invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0092] Alternatively, the processor can perform various functions of the electronic device by running or executing software programs stored in memory, and by calling data stored in memory.

[0093] In a specific implementation, as one example, the processor may include one or more CPUs. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include servers, desktop computers, laptops, smartphones, tablets, embedded computers, etc., where the embedded computer includes vehicles and robots, etc.

[0094] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0095] It should be noted that the structure of the electronic device shown in the accompanying drawings of this invention does not constitute a limitation thereof. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0096] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0097] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0098] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0099] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0100] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0103] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] To achieve the above-mentioned technical effects, the present invention proposes the following key technical points:

[0105] Key Point 1: Proposing a state-level bridge-circle decomposition mechanism. This invention uses the state risk sensitivity function as the core representation quantity and divides the state set into bridge and circle domains based on a threshold. The bridge domain corresponds to highly sensitive states, focusing on suppressing error propagation; the circle domain corresponds to low-sensitive states, focusing on maintaining distribution elasticity. This decomposition allows for the application of supervisory constraints to the same sequence based on state differences, improving the consistency between training and inference. This mechanism improves the effectiveness of supervisory allocation without increasing the complexity of the model structure.

[0106] Key Point 2: A joint optimization mechanism combining dynamic hybrid supervision and risk-guided correction is proposed. This invention employs a unified hybrid objective, dynamically fusing hard and soft supervision according to state; and combines confidence and entropy strategies to complete weight updates. Furthermore, a risk correction term is introduced to enhance the interval stability of key states and reduce the risk of cascading shifts. Thus, steady-state convergence and later refinement can be achieved within the same training cycle, avoiding target drift.

[0107] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0108] This invention adopts a unified process of "state unfolding—risk partitioning—hybrid optimization—deployment verification". This process connects data, training and evaluation modules through a unified interface, which facilitates engineering orchestration.

[0109] S1. State Unfolding: Construct a prefix state sequence in autoregressive order and save the context and task identifier. The state information is also used for subsequent partitioning and error statistics.

[0110] S2. Supervised Acquisition: For each state, acquire the probability distribution (soft label), hard label, confidence score, and entropy statistics of the teacher model. The supervised acquisition phase can be executed offline in parallel to reduce online overhead.

[0111] S3. Risk Partitioning: Calculate state risk and divide the region into bridge and circle domains, forming partition labels. Partition labels can be reused for training weighting and evaluation slice analysis.

[0112] S4. Hybrid Training: Soft and hard supervision are jointly computed at the state level and dynamically weighted. The joint optimization process maintains a single model parameter space solution, reducing process complexity.

[0113] S5. Deployment Evaluation: Select the final model based on a comprehensive evaluation of accuracy, stability, latency, and cost, and generate an auditable evaluation report. During the evaluation phase, it is recommended to use a unified benchmark and a fixed random seed to improve comparability.

[0114] To more clearly demonstrate the practicality and feasibility of the above steps, the following example, model distillation in a natural language question-answering scenario, will explain in detail the specific execution method of each step. In this example, the teacher model is LLaMA2-7B, the student model is MobileBERT, the input is a natural language medical question, and the output is the corresponding answer text.

[0115] In some embodiments of the present invention, step S1 includes the following detailed implementation points:

[0116] S11. Perform state expansion on the samples to keep the training and inference decoding directions consistent.

[0117] For example, consider a training sample where the user question is "What are the main side effects of aspirin?", and the corresponding standard answer sequence is "Common side effects of aspirin include gastrointestinal bleeding, tinnitus, and stomach ulcers." Following the autoregressive order, each token in the answer sequence is expanded into a prefix state sequence, as shown in the table below:

[0118]

[0119] Therefore, the state sequence of this sample is obtained as S = [s1, s2, ..., s]. T (T is the length of the answer sequence). This state sequence will be directly used for risk partitioning in S3 and mixed training in S4. This constraint can effectively avoid the risk of information leakage during the training phase.

[0120] S12. Establish a reusable supervised cache to support breakpoint resume training and experiment reproduction.

[0121] For each state s in the above state sequence S t Perform the following two pre-calculations respectively:

[0122] (i) Call the teacher model and calculate the probability distribution of the teacher model for each candidate word in the vocabulary in the current state. The specific calculation process for this probability distribution is as follows: Given state s... t The generated prefix sequence is fed into the teacher model as input. After passing through layers of neural network operations, the teacher model generates a raw score (logit value) for each candidate word in the vocabulary at the output layer. Then, a softmax normalization operation is applied to these raw scores (i.e., taking the exponent of each score and dividing by the sum of the exponents of all scores), converting the raw scores into probability values ​​between 0 and 1, with the sum of the probabilities of all candidate words being 1. The resulting probability distribution... This represents the probability that the teacher model considers the next word to be one of the candidate words in the vocabulary. For example, given the input "aspirin", the teacher model might output "common" with a probability of 0.70, "main" with a probability of 0.20, and "side effect" with a probability of 0.05 after the above calculation. This probability distribution is called the soft label. Simultaneously, the word with the highest probability from this distribution is taken as the hard label (i.e., the single prediction result that the teacher model is most confident in, "common" in the example above), denoted as a* = argmax. The probability distribution and hard labels of the aforementioned teacher model are stored as offline data, along with timestamps and model version numbers, in a cache file. Subsequent training cycles will directly read from the cache to avoid repeatedly calling the teacher model.

[0123] (ii) During the training process, the student model trains the same state s t It also performs forward inference, calculating the probability distribution of the student model itself in real time. , where θ are the trainable parameters of the student model. The probability distribution of the student model is dynamically calculated in each training step and changes as the parameters θ are updated, without the need for caching.

[0124] S13. Construct batches by bucketing according to length and task to improve throughput and reduce filling redundancy.

[0125] The state sequences of all samples in the training set are grouped by length: sequences ≤10 are placed in the small bucket, sequences 11-20 in the medium bucket, and sequences ≥21 in the large bucket. Each iteration samples only from one bucket and dynamically fills it to the maximum length within that bucket. Dynamic filling to the maximum length means that in each batch, the longest sequence in that batch is first found, and then the remaining shorter sequences in the batch are padded with placeholders to that length, ensuring all sequences in the same batch are of equal length to meet the requirements of parallel computing. For example, if a batch samples three sequences with lengths of 5, 8, and 10 from the small bucket (length ≤10), then sequences of length 5 and 8 are padded to length 10. Compared to traditional methods that uniformly fill all sequences to the global maximum length, the difference in sequence length within each bucket is smaller after bucketing, significantly reducing the computational waste during filling. In this example, this strategy improves the effective GPU computing power utilization by approximately 25%.

[0126] S21. Define the soft-supervised loss: This term measures the difference between the probability distribution of the student model and the probability distribution of the teacher model. Specifically, for each state s, the probability distribution of the teacher model is calculated. Probability distribution of the student model The KL divergence (Kullback-Leibler divergence, a standard measure of the difference between two probability distributions) is calculated as follows: θ represents the trainable parameters of the student model, and s represents the current state (i.e., the input context).

[0127] ;

[0128] Where D(·||·) represents the KL divergence, used to measure the difference between two probability distributions. The smaller this loss value, the closer the student model is to the teacher model.

[0129] For example, for state s3 (prefix "aspirin"), the teacher output distribution is {"common": 0.70, "primary": 0.20, "side effects": 0.05, others: 0.05}, then the soft-supervised loss calculates the KL divergence between the student distribution and the teacher distribution.

[0130] S22. Define Hard Supervision Loss: Unlike soft supervision loss, which focuses on the differences in the overall probability distribution, hard supervision loss only focuses on the prediction result that the teacher model is most confident about (i.e., the hard label a*), requiring the student model to give the highest possible prediction probability for that word. Specifically, hard supervision loss is the negative log-likelihood of the student model for the hard label a*. The smaller the loss value, the better the student model aligns with the teacher model's target at key points. In short, soft-supervised loss allows the student model to learn the teacher's complete probability distribution, while hard-supervised loss allows the student model to focus on learning the teacher's most certain predictions.

[0131]

[0132] π_θ(a*|s) represents the probability that the student model predicts the hard label a*, θ is the student model parameter, and s is the current state; for state s3, a* = "common", and the hard supervision loss is the negative log-likelihood of the student model predicting "common".

[0133] S23. The two types of supervision signals mentioned above are functionally complementary: soft supervision loss (S21) allows the student model to mimic the probability distribution of the teacher model across all candidate words as completely as possible, preserving the diversity and detailed information of the distribution; hard supervision loss (S22) allows the student model to concentrate its prediction probabilities on the words most certain by the teacher, ensuring the accuracy of predictions at key positions. When combined, the student model can closely approximate the teacher's overall distribution without bias at key decision positions. The specific fusion method of the two types of losses is implemented through a dynamic weighting mechanism in the subsequent step S4.

[0134] In some embodiments of the present invention, step S3 includes: dividing all states into high-risk bridge domains and low-risk circle domains by calculating the risk sensitivity of each state and comparing it with a threshold, specifically including the following sub-steps.

[0135] S31. Define State Risk Aggregation: First, define a word-level local risk sensitivity κ(a|s), which measures the additional loss risk to the subsequently generated sequence caused by the student model's prediction of word a deviating from the teacher model in state s. Specifically, κ(a|s) can be calculated by the difference between the teacher model's probability and the student model's probability: κ(a|s) = π_T(a|s) · |π_T(a|s) - π_θ(a|s)|, where π_T(a|s) is the teacher model's prediction probability of word a, and π_θ(a|s) is the student model's prediction probability of word a. This formula means that the more a word the teacher model values ​​(the larger π_T(a|s) is), and the greater the prediction deviation between the student model and the teacher model on that word (the larger |π_T(a|s) - π_θ(a|s)| is, the higher the risk sensitivity of that word. The larger the value of κ(a|s), the more severe the consequences of the word deviation. Then, for each state s, the maximum value of the local risk sensitivity of all candidate words in that state is calculated as the overall risk measure for that state.

[0136] Where: s represents the current state (input context), V represents the vocabulary (i.e., the set of all candidate words that the model can choose), a represents a candidate word in the vocabulary, κ(a|s) represents the local risk sensitivity brought about by choosing word a in state s, and κ(s) is the state-level risk measure obtained by taking the maximum value of all candidate words. The larger the value of κ(s), the more sensitive the state is to prediction bias, and the higher the risk.

[0137] S32. Define the Bridge Region and the Garden Region: Based on the state risk metric κ(s) mentioned above, divide all states into two regions. The core function of this division is that high-risk states in the Bridge Region will be subject to harder supervision in subsequent step S4 (i.e., relying more on the teacher's most certain prediction to constrain the student), while low-risk states in the Garden Region will be subject to softer supervision (i.e., preserving more of the diversity of the teacher's probability distribution), thereby achieving differentiated training strategies for different risk regions. The division rule can be stably implemented through validation set threshold calibration.

[0138] ;

[0139] In the above formula, B represents the Bridge region, which is the set of states where the risk metric κ(s) is greater than the threshold τ. These states are highly sensitive to prediction bias, requiring the next term to be accurate. G represents the Garden region, which is the set of states where the risk metric κ(s) is not greater than the threshold τ. These states have a higher tolerance for prediction bias, allowing for the flexible selection of multiple reasonable terms. τ is a preset risk threshold, which is a positive real number and can be determined on the validation set through grid search according to the specific task. A typical value range is between 0.1 and 0.9. The larger τ is, the fewer states are identified as Bridge regions; the smaller τ is, the more states are identified as Bridge regions.

[0140] S33. The bridge domain focuses on risk suppression, while the circular domain focuses on maintaining distribution elasticity. After partitioning, the error evolution can be statistically analyzed separately for the bridge domain and the circular domain.

[0141] In some embodiments of the present invention, step S4 includes: merging the soft supervision loss and hard supervision loss into a hybrid training objective according to the bridge domain-circle domain partitioning result, and introducing a risk-guided correction term to enhance the stability of bridge domain prediction, specifically including the following sub-steps.

[0142] S41. Define the hybrid objective: Combine the soft supervision loss defined in step S21. Compared with the hard supervision loss defined in step S22 The weighted summation is used to merge the results into a unified training objective. This hybrid objective is directly used to drive the gradient update of the student model parameters θ, and is the actual objective function optimized by the student model throughout the training process.

[0143]

[0144] The meanings of the symbols in the objective formula of the above hybrid loss function are as follows: Here, s represents the mixed supervision loss (i.e., the objective function that the student model actually optimizes), θ represents the current state, θ represents the student model parameters, and λ(s) represents the soft supervision weights (real numbers ranging from 0 to 1, which determine the respective proportions of soft and hard supervision). The soft supervision loss defined in step S21, The hard-supervised loss is defined for step S22. λ(s) is key to connecting step S3 (risk partitioning) and step S4 (hybrid training): for the garden area (low-risk) state, λ(s) takes a larger value, with soft supervision dominating in the hybrid objective, preserving the diversity of teacher distribution; for the bridge area (high-risk) state, λ(s) takes a smaller value, with hard supervision dominating, ensuring the prediction accuracy of key locations. The following are two specific calculation methods for λ(s):

[0145] S42. Confidence weighting: The first is confidence weighting. The teacher model is used to judge whether the state belongs to the bridge domain or the garden domain based on the prediction probability of the most likely token: When the prediction of the teacher model for a certain state is highly concentrated on a certain token (the maximum probability is close to 1), it indicates that this state belongs to the bridge domain and hard supervision should be emphasized. Based on this, the soft supervision weight is defined as:

[0146]

[0147] In the above formula, is the soft supervision weight based on confidence, is the probability that the teacher model predicts the token in state s, is the maximum probability among all candidate tokens predicted by the teacher model. When the teacher model is highly confident (the maximum probability is close to 1), , the hybrid objective emphasizes hard supervision; when the teacher model is uncertain (the probabilities are relatively scattered), is close to 1, and the hybrid objective emphasizes soft supervision. For example, for state s2 (prefix "aspirin", the probability of the teacher's most likely token "of" is 0.95), = 1 - 0.95 = 0.05, and the hard supervision weight in the hybrid objective is 0.95, almost completely relying on hard label constraints.

[0148] S43. Entropy weighting: The second is entropy weighting. The information entropy of the teacher model's prediction distribution is used to measure the degree of uncertainty: When the prediction probabilities of the teacher model are scattered among multiple tokens (the entropy is large), it indicates that there are multiple reasonable choices at this position, belonging to the garden domain, and soft supervision should be emphasized. Based on this, the soft supervision weight is defined as:

[0149]

[0150] In the above formula, is the soft supervision weight based on entropy, is the prediction entropy of the teacher model in state s (i.e., the uncertainty measure of the probability distribution), is the vocabulary size (i.e., the total number of candidate tokens), is the maximum possible entropy value (used for normalization), is the probability that the teacher model predicts token a in state s, and the sum traverses all candidate tokens in the vocabulary. When the teacher distribution is flat (the entropy is large), is close to 1, emphasizing soft supervision; when the teacher distribution is sharp (the entropy is small), is close to 0, emphasizing hard supervision.

[0151] Among them, the prediction entropy That is, the classic Shannon entropy in information theory, which is directly calculated by summing the softmax probability distributions output by the teacher model for all candidate tokens in the vocabulary V at state s according to the above formula, is used to characterize the uncertainty of the prediction in that state. The larger the entropy, the flatter the teacher distribution and the more uncertain the prediction; the smaller the entropy, the sharper the distribution and the more confident the prediction.

[0152] Based on the above mixed objective, a risk-guided correction term can be further introduced to replace the original hard supervision loss with the corrected hard supervision loss to enhance the prediction interval stability of the key states in the bridge domain:

[0153]

[0154]

[0155] In the above risk-guided correction formula, is the corrected hard supervision loss, which adds a correction term to the original hard supervision loss In this correction term: α is a positive correction intensity coefficient (a hyperparameter that controls the correction strength), is the log probability interval of the student model at state s and hard label a*, (·|s) is the logit output value of the student model for each token at state s (i.e., the original score before softmax normalization), represents the new state formed by appending the hard label a* after state s, V is the vocabulary, and the summation traverses all candidate tokens in the vocabulary. The role of this correction term is: at the key states in the bridge domain, when the prediction probability of the student model for the hard label is not large enough compared to the probabilities of other tokens, an additional penalty is imposed to force the student model to widen the gap between the correct prediction and the wrong prediction, thereby enhancing the stability of the bridge domain prediction and preventing small deviations from accumulating and amplifying in subsequent generations.

[0156] Combining the above steps S41 to S43, taking the aspirin Q&A as an example below, the specific operation process of the above formula in a complete training step is illustrated. Assume that the current training step processes the state (prefix "aspirin", candidate token "的"): First step, read the probability distribution of the teacher model at and the hard label a* = "的" (the pre-computed result of step S12) from the cache; Second step, the student model performs forward inference on and calculates the student probability distribution in real time.; In the third step, calculate the soft supervision loss (KL divergence between the teacher distribution and the student distribution) and the hard supervision loss (negative log-likelihood of the student predicting "de") corresponding to steps S21 and S22 respectively; In the fourth step, calculate the soft supervision weight λ(s2) of this state by step S42 or S43. For example, when using confidence weighting, the probability of the teacher on "de" is 0.95, then λ conf (s2)=1 - 0.95 = 0.05, which means this state belongs to the bridge domain and the hard supervision accounts for 95% of the weight; In the fifth step, substitute λ(s2) into the mixed objective formula l hyb =0.05×l soft +0.95×l hard , and obtain the final loss value of this state (step S41); In the sixth step, use this loss value to calculate the gradient through the backpropagation algorithm and update the student model parameters θ. On the bridge domain state, the risk-guided correction term can also be used to replace to enhance the prediction stability. Repeat the above six steps for all states in all training samples. After multiple rounds of training iterations, the student model gradually learns the knowledge of the teacher model.

[0157] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in cooperation with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. To avoid repetition, they will not be elaborated here. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0158] As Figure 2 shown, the present invention also proposes a hybrid supervision language model distillation device B based on bridge domain - garden domain decomposition, which includes:

[0159] An initial module, which obtains a natural language Q&A data set. The natural language Q&A data set includes multiple Q&A samples, and each Q&A sample includes a question and its corresponding standard answer; According to the autoregressive order, expand the standard answer token by token into a prefix state sequence S = [s1, s2,..., s T , where T is the sequence length of the standard answer;

[0160] A prediction module, for each state s t in the prefix state sequence S, call the teacher model and the student model to calculate their probability distributions and hard labels a* respectively, and obtain the teacher distribution, the teacher label, and the student distribution;

[0161] A loss module, which divides all the prefix state sequences S into small buckets, medium buckets, and large buckets according to the length, and only samples from one bucket in each iteration; According to the approximation degree of the student distribution to the teacher distribution, construct a soft loss function, and according to the negative log-likelihood of the student model for the hard label a*, construct a hard loss function;

[0162] The training module uses the difference between the probability distributions of the teacher model and the student model in the same state as the word-level local risk sensitivity. It determines whether the local risk sensitivity is greater than a preset value. If it is, it's a bridge region; the hard loss function is multiplied by weight A and then summed with the soft loss function to form a hybrid loss function. Otherwise, it's a circle region; the hard loss function is multiplied by weight B and then summed with the soft loss function to form a hybrid loss function. Weight A is greater than weight B. The hybrid loss function is used to calculate the gradient through backpropagation and update the student model. The prediction module, loss module, and training module are iteratively executed repeatedly until the hybrid loss function converges, and the current student model is saved as the final model.

[0163] The question-and-answer module takes the question text as input into the final model and outputs the answer to that question text.

[0164] The aforementioned hybrid supervised language model distillation device based on bridge-circle domain decomposition, wherein the training module includes:

[0165] Calculate the local risk sensitivity :

[0166]

[0167] Let κ(a|s) represent the candidate lexical units in the vocabulary V, and let κ(a|s) represent the local risk sensitivity resulting from choosing lexical unit a in state s.

[0168] Based on risk aggregation Delineating bridge domains With the Garden Area ;

[0169] ;

[0170] τ is a preset risk threshold, with a value ranging from 0.1 to 0.9;

[0171] The hybrid loss function :

[0172] ;

[0173] ;

[0174] θ represents the student model parameters, and λ(s) represents the soft supervision weights. It is a soft loss function. It is a hard loss function; Predicting words for the teacher model in state s The probability, This represents the maximum probability among all candidate lexical units in the teacher model.

[0175] The aforementioned hybrid supervised language model distillation device based on bridge-circle domain decomposition, wherein the training module includes:

[0176] Calculate the local risk sensitivity ;

[0177] Based on risk aggregation Delineating bridge domains With the Garden Area ;

[0178]

[0179] The hybrid loss function :

[0180] ;

[0181] ;

[0182] ;

[0183] .

[0184] In the formula The prediction entropy of the teacher model in state s. For vocabulary size, The maximum possible entropy value; This represents the corrected hard loss function; α is a positive correction strength coefficient. Let be the log-probability interval of the student model at state s and hard label a*. (·|s) represents the logit output value of the student model for each word in state s. This indicates the new state formed by appending the hard label a* to state s.

[0185] The present invention also proposes a client for implementing any of the hybrid supervised language model distillation devices based on bridge domain-circle domain decomposition.

[0186] like Figure 3 As shown, in another embodiment of the present invention, a first electronic device A is also proposed, including the aforementioned distillation device B for a hybrid supervised language model based on bridge domain-circle domain decomposition.

[0187] like Figure 4 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D via wired or wireless information transmission schemes. The data acquisition device C is used to collect and obtain text questions, and the information display device D is used to display the answer results obtained by the analysis of the present invention.

[0188] The information display device D can process and organize the data output by the first electronic device A based on an information display mechanism to improve the readability of the data. This information display mechanism can be manually preset, for example, visualizing the data output by the first electronic device A. It can present the user with the specified key information based on user-defined display parameters and / or attributes, such as the data range and font, color, and scrolling options. Users can access this information more quickly without needing to navigate to secondary pages or scroll through pages, saving them time and effort. Alternatively, the information display mechanism can be an artificial intelligence (AI) display model that learns the user's key information interests based on past usage habits, such as viewing time, click count, and edit count, and automatically presents rich and necessary key information.

[0189] The present invention also provides a computer program product, which includes a computer program that can be stored on a readable storage medium. When the computer program is executed by a processor, the computer is able to execute the hybrid supervised language model distillation method based on bridge domain-circle domain decomposition provided by the above methods.

[0190] In another embodiment, the present invention also proposes a storage medium VIII for storing a computer program that executes the hybrid supervised language model distillation method based on bridge-domain-circle decomposition. It should be understood that the storage medium in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0191] Figure 5 A schematic block diagram of a second electronic device 1000 that can be used to implement embodiments of the present invention is shown. The second electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0192] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from storage medium VIII into random access memory (RAM) III. The RAM III may also store various programs and data required for the operation of the device 1000. The computing unit I, ROM II, and RAM III are interconnected via bus IV. An input / output (I / O) interface V is also connected to bus IV.

[0193] Multiple components in the second electronic device 1000 are connected to I / O interface V, including: input unit VI, such as a keyboard, mouse, etc.; output unit VII, such as various types of displays, speakers, etc.; storage medium VIII, such as a disk, optical disk, etc.; and communication unit IX, such as a network card, modem, wireless transceiver, etc. Communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0194] The computing unit I can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S5. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, computing unit I can be configured to perform methods by any other suitable means (e.g., by means of firmware).

[0195] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A method for distilling a hybrid supervised language model based on bridge domain-circle domain decomposition, characterized in that, include: The initial step involves acquiring a natural language question-and-answer dataset, which includes multiple question-and-answer samples. Each sample contains a question and its corresponding standard answer. Following an autoregressive process, the standard answer is expanded into a prefix state sequence S = [s1, s2, ..., s...] token by token. T ], where T is the sequence length of the standard answer; The prediction step involves processing each state s in the prefix state sequence S. t The teacher model and student model are called to calculate their probability distributions and hard labels a*, respectively, to obtain the teacher distribution, teacher labels and student distributions; The loss step involves dividing all prefix state sequences S into small, medium, and large buckets according to their length, and sampling from only one bucket in each iteration; a soft loss function is constructed based on the approximation of the student distribution to the teacher distribution, and a hard loss function is constructed based on the negative log-likelihood of the student model to the hard label a*. The training steps involve using the difference between the probability distributions of the teacher model and the student model in the same state as the word-level local risk sensitivity. If the local risk sensitivity is greater than a preset value, it's considered a bridge region. The hard loss function is multiplied by weight A and then summed with the soft loss function to form a hybrid loss function. Otherwise, it's considered a circle region. The hard loss function is multiplied by weight B and then summed with the soft loss function to form a hybrid loss function. Weight A is greater than weight B. The hybrid loss function is used to calculate the gradient through backpropagation and update the student model. The prediction, loss, and training steps are iterated repeatedly until the hybrid loss function converges. The current student model is then saved as the final model. The question-and-answer process involves inputting the question text into the final model to obtain the answer to the question text.

2. The distillation method for hybrid supervised language models based on bridge domain-circle domain decomposition as described in claim 1, characterized in that, The training steps include: Calculate the local risk sensitivity : Let κ(a|s) represent the candidate lexical units in the vocabulary V, and let κ(a|s) represent the local risk sensitivity resulting from choosing lexical unit a in state s. Based on risk aggregation Delineating bridge domains With the Garden Area ; ； τ is a preset risk threshold, with a value ranging from 0.1 to 0.9; The hybrid loss function : ； ； θ represents the student model parameters, and λ(s) represents the soft supervision weights. It is a soft loss function. It is a hard loss function; Predicting words for the teacher model in state s The probability, This represents the maximum probability among all candidate lexical units in the teacher model.

3. The distillation method for hybrid supervised language models based on bridge domain-circle domain decomposition as described in claim 2, characterized in that, The training steps include: Calculate the local risk sensitivity ; Based on risk aggregation Delineating bridge domains With the Garden Area ; The hybrid loss function : ； ； ； . In the formula The prediction entropy of the teacher model in state s. For vocabulary size, The maximum possible entropy value; This represents the corrected hard loss function; α is a positive correction strength coefficient. Let be the log-probability interval of the student model at state s and hard label a*. (·|s) represents the logit output value of the student model for each word in state s. This indicates the new state formed by appending the hard label a* to state s.

4. A distillation apparatus for a hybrid supervised language model based on bridge domain-circle domain decomposition, characterized in that, include: The initial module acquires a natural language question-and-answer dataset, which includes multiple question-and-answer samples. Each sample contains a question and its corresponding standard answer. Following an autoregressive order, the standard answer is expanded into a prefix state sequence S = [s1, s2, ..., s...]. T ], where T is the sequence length of the standard answer; The prediction module, for each state s in the prefix state sequence S t The teacher model and student model are called to calculate their probability distributions and hard labels a*, respectively, to obtain the teacher distribution, teacher labels and student distributions; The loss module divides all prefix state sequences S into small, medium, and large buckets according to their length, and samples from only one bucket in each iteration; it constructs a soft loss function based on how closely the student distribution approximates the teacher distribution, and a hard loss function based on the negative log-likelihood of the student model with respect to the hard label a*. The training module uses the difference between the probability distributions of the teacher model and the student model in the same state as the word-level local risk sensitivity. It determines whether the local risk sensitivity is greater than a preset value. If it is, it's a bridge region; the hard loss function is multiplied by weight A and then summed with the soft loss function to form a hybrid loss function. Otherwise, it's a circle region; the hard loss function is multiplied by weight B and then summed with the soft loss function to form a hybrid loss function. Weight A is greater than weight B. The hybrid loss function is used to calculate the gradient through backpropagation and update the student model. The prediction module, loss module, and training module are iteratively executed repeatedly until the hybrid loss function converges, and the current student model is saved as the final model. The question-and-answer module takes the question text as input into the final model and outputs the answer to that question text.

5. The distillation apparatus for a hybrid supervised language model based on bridge domain-circle domain decomposition as described in claim 4, characterized in that, This training module includes: Calculate the local risk sensitivity : Let κ(a|s) represent the candidate lexical units in the vocabulary V, and let κ(a|s) represent the local risk sensitivity resulting from choosing lexical unit a in state s. Based on risk aggregation Delineating bridge domains With the Garden Area ; ； τ is a preset risk threshold, with a value ranging from 0.1 to 0.9; The hybrid loss function : ； ； θ represents the student model parameters, and λ(s) represents the soft supervision weights. It is a soft loss function. It is a hard loss function; Predicting words for the teacher model in state s The probability, This represents the maximum probability among all candidate lexical units in the teacher model.

6. The distillation apparatus for a hybrid supervised language model based on bridge-circle decomposition as described in claim 5, characterized in that, This training module includes: Calculate the local risk sensitivity ; Based on risk aggregation Delineating bridge domains With the Garden Area ; The hybrid loss function : ； ； ； . In the formula The prediction entropy of the teacher model in state s. For vocabulary size, The maximum possible entropy value; This represents the corrected hard loss function; α is a positive correction strength coefficient. Let be the log-probability interval of the student model at state s and hard label a*. (·|s) represents the logit output value of the student model for each word in state s. This indicates the new state formed by appending the hard label a* to state s.

7. A client for implementing any of the hybrid supervised language model distillation apparatuses based on bridge domain-circle domain decomposition as described in claims 4-6.

8. An electronic device, characterized in that, The device includes a hybrid supervised language model distillation apparatus based on bridge domain-circle domain decomposition as described in claims 4-6. The electronic device may be connected to an information display device, which is used to display the answer result with user-set display parameters, attributes, or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the hybrid supervised language model distillation method based on bridge-circle decomposition as described in any one of claims 1-3.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the hybrid supervised language model distillation method based on bridge domain-circle domain decomposition as described in any of claims 1-3.

Images