Text information answering method and device

Abstract

The application discloses a text information reply method and device, and relates to the technical field of knowledge distillation, which comprises the following steps: based on the fine-grained distribution and coarse-grained structure of the attention layer, performing full-dimension knowledge migration, and combining attention grouping integration, learnable matrix dimension unification and double loss constraints to perform corresponding text processing and other operations, solving the technical problem that in the related art, the knowledge migration of distillation is one-sided, when the teacher network and the student network perform one-to-one knowledge distillation between layers, the number and dimension of the attention maps generated by both parties in the corresponding distillation layer cannot be matched, greatly affecting the distillation effect and the performance of the student network, and the technical effect of significantly improving the efficiency and effect of knowledge distillation, helping the student network to fully absorb the attention mode of the teacher network and enhancing the performance of the student network in text processing and other tasks.

Description

Technical Field

[0001] This application relates to the field of knowledge distillation technology, and in particular to a method and apparatus for responding to textual information. Background Technology

[0002] In the multi-head attention mechanism of the Transformer model, the teacher network usually has more attention heads than the student network in order to capture richer feature association information. However, the difference in the number of heads directly leads to a mismatch in the number of attention outputs between the teacher and student networks. This makes it difficult to directly apply the "one-to-one" attention output matching and comparison logic in traditional distillation methods, and it is impossible to effectively construct a distillation loss function. Consequently, it hinders the efficient advancement of the knowledge distillation process, making it difficult for the student network to fully learn and absorb the attention-level knowledge of the teacher network.

[0003] Currently, related technologies can leverage principal component analysis to overcome the limitations of traditional white-box distillation on model architecture, enabling knowledge distillation between teacher and student models with different hidden layer sizes and architectures, thus improving the compression ratio. Furthermore, the combination of two rounds of distillation can further improve the performance of student models compared to a single distillation method.

[0004] However, the related technologies have not built targeted solutions around the multi-head attention mechanism of the Transformer architecture, but only focused on feature dimensionality reduction and output soft labeling, without deeply exploring the two core knowledge of fine-grained probability distribution and coarse-grained structural features contained in the attention mechanism under the Transformer architecture.

[0005] In summary, in related technologies, knowledge transfer through distillation is too one-sided. When teacher networks and student networks perform one-to-one knowledge distillation between layers, the number of attention maps generated by both parties in the corresponding distillation layer is often mismatched, which greatly affects the distillation effect and the performance of the student network and urgently needs to be solved. Summary of the Invention

[0006] This application provides a text information response method and apparatus to at least solve the technical problem in the related art that the knowledge transfer distillation is too one-sided, and when the teacher network and the student network perform one-to-one knowledge distillation between layers, the number and dimension of the attention maps generated by the two parties in the corresponding distillation layer cannot be matched, which greatly affects the distillation effect and the performance of the student network.

[0007] This application provides a text message response method, comprising the following steps: selecting a preset number of teacher network layers that meet a preset network layer contribution requirement from the teacher network of a preset natural language processing model, and selecting the preset number of student network layers that meet the preset network layer contribution requirement from the student network of the natural language processing model; performing network layer matching on the teacher network layers and the student network layers to construct a corresponding distillation layer set; obtaining attention map sets of the teacher network layers and student network layers in any distillation layer of the distillation layer set, and grouping and integrating the attention map sets to obtain attention map groups that meet a preset number requirement; and performing column-wise concatenation of the attention maps in the attention map groups to generate a corresponding... Teacher aggregation matrix and student aggregation matrix; determine the matrix dimensions corresponding to the teacher aggregation matrix and the student aggregation matrix respectively, so as to construct the learnable transformation matrix corresponding to the teacher aggregation matrix and the student aggregation matrix respectively according to the matrix dimensions, and perform matrix multiplication operation on the teacher aggregation matrix and the student aggregation matrix with the corresponding learnable transformation matrix respectively to obtain the corresponding teacher feature matrix and student feature matrix; based on the teacher feature matrix and the student feature matrix, determine the double distillation loss function corresponding to the natural language processing model, so as to train the student network based on the double distillation loss function, and use the trained student network to infer the text information input by the target user and generate the response information corresponding to the text information.

[0008] This application also provides a text message response device, comprising: a network layer matching module, configured to select a preset number of teacher network layers that meet a preset network layer contribution requirement from the teacher network of a preset natural language processing model, and select the preset number of student network layers that meet the preset network layer contribution requirement from the student network of the natural language processing model, and perform network layer matching on the teacher network layers and the student network layers to construct a corresponding distillation layer set; and a grouping and integration module, configured to obtain attention graph sets of the teacher network layers and student network layers in any distillation layer of the distillation layer set, and group and integrate the attention graph sets to obtain attention graph groups that meet a preset number requirement, and perform column-wise concatenation operation on the attention graphs in the attention graph groups to generate a corresponding teacher network layer. The system comprises a teacher aggregation matrix and a student aggregation matrix; a matrix transformation module, used to determine the matrix dimensions corresponding to the teacher aggregation matrix and the student aggregation matrix respectively, so as to construct the learnable transformation matrices corresponding to the teacher aggregation matrix and the student aggregation matrix respectively according to the matrix dimensions, and to perform matrix multiplication operations on the teacher aggregation matrix and the student aggregation matrix with the corresponding learnable transformation matrices respectively to obtain the corresponding teacher feature matrix and student feature matrix; and a text response module, used to determine the double distillation loss function corresponding to the natural language processing model based on the teacher feature matrix and the student feature matrix, so as to train the student network based on the double distillation loss function, and to use the trained student network to infer the text information input by the target user and generate the response information corresponding to the text information.

[0009] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described text message response methods.

[0010] This application also provides a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described text information response methods.

[0011] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described text message response methods.

[0012] This application allows for the selection of a predetermined number of teacher network layers that meet predetermined contribution requirements from the teacher network of a pre-defined natural language processing model, and a predetermined number of student network layers that meet predetermined contribution requirements from the student network of the same model. Network layer matching is then performed on the teacher and student network layers to construct corresponding distillation layer sets. Attention map sets for both teacher and student network layers within any distillation layer of the distillation layer set are obtained, and these sets are grouped and integrated to obtain attention map groups that meet predetermined contribution requirements. The attention maps within these groups are then concatenated column-wise to generate corresponding teacher and student aggregation matrices. The matrix dimensions corresponding to the teacher and student aggregation matrices are determined, and learnable transformation matrices are constructed based on these dimensions. Finally, the teacher and student aggregation matrices are analyzed separately. The aggregation matrix and the corresponding learnable transformation matrix are multiplied to obtain the corresponding teacher feature matrix and student feature matrix. Based on the teacher feature matrix and student feature matrix, the double distillation loss function corresponding to the natural language processing model is determined. The student network is trained based on the double distillation loss function, and the trained student network is used to infer the text information input by the target user and generate the corresponding response information. Therefore, this method can solve the technical problem in related technologies where knowledge transfer by distillation is too one-sided, and when the teacher network and student network perform one-to-one knowledge distillation between layers, the number and dimension of the attention maps generated by both parties in the corresponding distillation layer are mismatched, which greatly affects the distillation effect and the performance of the student network. This method significantly improves the efficiency and effect of knowledge distillation, helps the student network fully absorb the attention pattern of the teacher network, and enhances the performance of the student network in tasks such as text processing and image recognition. Attached Figure Description

[0013] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a text message response method provided according to an embodiment of this application;

[0015] Figure 2 A schematic diagram of a knowledge distillation process for attention graph groups is provided as an embodiment of this application;

[0016] Figure 3 This is an example diagram of a text message response device according to an embodiment of this application.

[0017] Among them, 10 is a text message response device, 100 is a network layer matching module, 200 is a group integration module, 300 is a matrix transformation module, and 400 is a text reply module. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0019] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0020] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] The specific application environment architecture or specific hardware architecture on which the text information response method depends is described here.

[0022] The embodiments of this application provide a text message response method.

[0023] like Figure 1 The diagram shown is a flowchart of a text message response method according to an embodiment of this application. The text message response method includes the following steps:

[0024] In step S101, a preset number of teacher network layers that meet the preset network layer contribution requirements are selected from the teacher network of the preset natural language processing model, and a preset number of student network layers that meet the preset network layer contribution requirements are selected from the student network of the natural language processing model. The teacher network layers and student network layers are matched to construct the corresponding distillation layer set.

[0025] Those skilled in the art should understand that, in current AI application scenarios, while neural network models based on the Transformer architecture have excellent performance, they suffer from significant resource consumption issues. This not only raises hardware performance requirements and increases the hardware and maintenance costs of model deployment, but also severely limits their promotion in resource-constrained scenarios such as mobile devices and edge computing terminals, making it difficult to meet the needs of lightweight deployment.

[0026] To address the aforementioned lightweighting challenges, related techniques can extract key knowledge from high-performance but structurally complex teacher networks and transfer it to simplified student networks. This significantly reduces model storage and computational overhead while ensuring that the performance of the student network is close to that of the teacher network. However, when this method is applied to Transformer models, the teacher network typically has more attention heads than the student network in the Transformer's multi-head attention mechanism (to capture richer feature associations). This mismatch in the number of attention outputs between the two networks renders the traditional "one-to-one" output matching and comparison logic of distillation ineffective, hindering the efficient construction of the distillation loss function and impeding the efficient progress of knowledge distillation. Consequently, the student network struggles to fully absorb the attentional knowledge from the teacher network.

[0027] Therefore, in terms of technical structure, the attention maps of the teacher network and the student network can be divided into the same number of corresponding groups (i.e., attention map groups) in the embodiments of this application, and the attention maps within the attention map groups can be spliced ​​together. Furthermore, by introducing a learnable matrix, the spliced ​​attention feature matrices (i.e., aggregation matrices) with different dimensions can be converted into square matrices of the same dimension to solve the problem of inconsistent dimensions caused by the difference in the number of multi-head attention, thus laying a solid foundation for subsequent loss calculation.

[0028] Furthermore, in terms of loss function design, this embodiment can adopt a dual-distillation loss with collaborative constraints. The first distillation loss can be processed by Softmax normalization and combined with the KL (Kullback-Leibler, relative entropy) divergence method to accurately quantify the fine-grained differences in the attention probability distribution between the teacher network and the student network. The second distillation loss is processed by max pooling downsampling, mean thresholding binarization, and XOR operation to capture the consistency differences in the macro-structure of attention. Subsequently, this embodiment can perform a weighted summation operation on the two losses to merge them into the total loss within the group, and form single-layer and global distillation losses through weighted aggregation to achieve comprehensive knowledge alignment between fine-grained distribution and coarse-grained structure.

[0029] In actual implementation, embodiments of this application can first select a preset number of corresponding layers from the teacher network and student network of the Transformer architecture of the natural language processing model to construct a one-to-one distillation layer set.

[0030] Therefore, the embodiments of this application construct a set of distillation layers by selecting teacher network layers and student network layers and performing network layer matching, thereby providing solid and reliable data support for the subsequent calculation of double distillation loss.

[0031] Optionally, in one embodiment of this application, before selecting a preset number of teacher network layers that meet the preset network layer contribution requirements from the teacher network of the preset natural language processing model, the method further includes: collecting historical text input data and historical response information corresponding to the natural language processing model; determining the network layer contribution of different network layers corresponding to the natural language processing model based on the historical text input data and historical response information, and calculating the relative criticality index of different network layers in the teacher network and student network according to the network layer contribution; comparing the relative criticality index with the preset criticality threshold to determine the number of indicators of relative criticality index that are greater than the criticality threshold, and determining the preset number based on the number of indicators.

[0032] The embodiments of this application can achieve efficient transfer of teacher knowledge to the student network in scenarios where the number of attention maps in the distillation layers corresponding to the teacher network and the student network is inconsistent, thus ensuring the effectiveness and reliability of knowledge distillation.

[0033] As one possible approach, embodiments of this application may assume that the total number of layers in the teacher network is a certain quantity. The total number of layers in the student network is another number. In this embodiment of the application, a certain number (i.e., a preset number) can be selected from both the teacher network and the student network. The network layers are used for "one-to-one" knowledge distillation. The number of network layers selected does not exceed the total number of layers in the student network, nor does it exceed the total number of layers in the teacher network. , .

[0034] Specifically, the process for determining the number of network layers that meet the preset network layer contribution requirements from the teacher network and student network in this embodiment is as follows:

[0035] 1. Data Acquisition Phase:

[0036] In the embodiments of this application, this stage requires the systematic collection of historical interaction data of the natural language processing model, which includes not only historical text input data of the user (such as dialogue query, text generation requirements, translation source text, etc.), but also historical response information output by the corresponding model (such as dialogue response, generated text, translation results, etc.).

[0037] It should be noted that, in the data collection process of this application embodiment, the integrity (covering different application scenarios of the model), authenticity (no manual tampering) and timeliness (reflecting the recent running status of the model) of the data must be ensured. At the same time, invalid data (such as garbled text and duplicate input) must be filtered out to lay a high-quality data foundation for subsequent analysis.

[0038] 2. Network layer contribution calculation stage:

[0039] This application's embodiments can quantify the contribution of different network layers (such as the encoder, decoder, attention, and fully connected layers of a Transformer) in an NLP (Natural Language Processing) model based on collected historical data and through specific evaluation methods (such as gradient attribution, attention weight analysis, and model ablation experiments). For example, by removing a network layer one by one through ablation experiments, the decline in the model's performance (such as accuracy and perplexity) on historical data is compared; the more significant the decline, the higher the contribution of that layer. Furthermore, by combining the feature correlations of historical input and output data, the actual contribution of each network layer to the model's output results can be determined.

[0040] 3. Calculation stage of relative criticality index:

[0041] After obtaining the contribution of each network layer, this embodiment of the application can calculate the relative criticality index of different network layers for the teacher network (usually a complex, high-precision pre-trained model) and the student network (usually a model to be compressed and lightweight). The calculation logic needs to combine the structural correspondence between the two network layers (such as the mapping relationship between the teacher layer and the student layer), with the contribution as the core, and introduce auxiliary factors such as layer dimension and parameter scale, and finally obtain the relative criticality index that reflects the importance of a certain layer to the overall performance of its own network, so as to realize the comparable analysis of the criticality of the teacher and student network layers.

[0042] 4. Preset quantity determination stage:

[0043] In this stage, the embodiments of this application first set a criticality threshold (for example, setting "contributing more than 15% to network performance" as the threshold) based on the model application scenario (such as edge deployment, real-time inference) and performance requirements (such as accuracy loss threshold, inference speed target). Then, the relative criticality indicators of each layer in the teacher and student networks are compared with the threshold to count the number of indicators (i.e., the total number of network layers with criticality exceeding the threshold). Finally, based on this number, the preset number in model compression or optimization is determined (such as the number of key network layers that need to be retained during model distillation).

[0044] Therefore, the embodiments of this application can determine the criticality of network layers through data-driven methods, thereby accurately locating the core layers of the model, providing a basis for model compression and distillation, and reducing redundant calculations; at the same time, it ensures that the student network retains key performance, balances model lightweighting and accuracy, and improves deployment efficiency.

[0045] Furthermore, embodiments of this application may be adapted according to the selected... A set of distillation layers is constructed based on a one-to-one correspondence between the layers, and any one of the distillation layers in the set is analyzed. In this embodiment, the distillation layer can be defined as... Where the range of values ​​for j is (That is, j can be 1 to 1) Any integer between these ranges corresponds to a different "one-to-one" layer.

[0046] It should be noted that, since all "one-to-one" corresponding layers follow the same processing logic and principles during distillation, therefore, for The derived conclusions, methods, and formulas are applicable to all other selected "one-to-one" corresponding distillation layers. There is no need to repeat the derivation for each distillation layer separately. By analyzing a single layer, the results can be generalized to all "one-to-one" distillation layers participating in the distillation.

[0047] In step S102, attention graph sets of the teacher network layer and student network layer in any distillation layer of the distillation layer set are obtained respectively, and the attention graph sets are grouped and integrated to obtain attention graph groups that meet the preset quantity requirements. The attention graphs in the attention graph groups are then spliced ​​column by column to generate the corresponding teacher aggregation matrix and student aggregation matrix.

[0048] Furthermore, in this embodiment of the application, for any distillation layer in the distillation layer set, the attention graph set of that layer in the teacher network (i.e., any distillation layer) and the attention graph set of that layer in the student network (i.e., any distillation layer) can be grouped and integrated to form an equal number of attention graph groups, and the attention graphs in each attention graph group are spliced ​​column by column to obtain the corresponding teacher aggregation matrix and student aggregation matrix.

[0049] Optionally, in one embodiment of this application, attention graph sets of the teacher network layer and student network layer in any distillation layer of the distillation layer set are obtained respectively, and the attention graph sets are grouped and integrated to obtain attention graph groups that meet the preset quantity requirements. This includes: generating multiple corresponding teacher layer attention graphs based on the multi-head attention mechanism in the teacher network layer, and generating multiple corresponding student layer attention graphs using the multi-head attention mechanism in the student network layer; determining the generation sequence number corresponding to the multiple teacher layer attention graphs and the multiple student layer attention graphs respectively, and dividing the multiple teacher layer attention graphs and the multiple student layer attention graphs into corresponding attention graph groups according to the generation sequence number to obtain attention graph groups that meet the preset quantity requirements.

[0050] It should be noted that the "one-to-one" distillation layer selected above... The embodiments of this application can define a multi-head attention mechanism for the teacher network layer, including... The size of the image can correspondingly generate a teacher-level attention map. , , ..., In student networks, the number of heads in the multi-head attention mechanism is... This can correspondingly generate student-level attention maps. , , ..., Generally speaking, .

[0051] In the knowledge distillation process, to establish the correspondence between the teacher's network attention graph and the student's network attention graph, the embodiments of this application first need to convert the teacher's network attention graph into a corresponding network attention graph. Attention maps, student networks Each attention map is divided into an equal number of... Attention map groups (i.e., the number of teacher groups and student groups are both 1) This allows for the indirect correspondence between teacher and student attention maps, using "groups" as intermediate units.

[0052] As one possible approach, embodiments of this application may employ a grouping strategy of "average distribution according to the order of attention map generation numbers," that is, for the teacher network... Each attention map is numbered from 1 to 1 in the order of its generation. The items are allocated sequentially to Group; for student networks Each attention map is assigned to a specific group according to the same sequence number allocation rule. Group.

[0053] It should be noted that the core objective of the grouping strategy in this application embodiment is to ensure that the number of teachers and students in each group is strictly the same (both are...). (That is, meeting the preset quantity requirements), subsequent knowledge transfer can be achieved through alignment processing of attention maps within the group (such as feature transformation, loss calculation), and the embodiments of this application are also applicable to The extreme scenario (i.e., all attention maps of the teacher and students are grouped into one group).

[0054] Therefore, the embodiments of this application allocate the generation sequence number of attention maps to each group in sequence, so that the number of attention maps contained in each group is as equal as possible, ensuring that the grouping logic is clear and orderly, and providing a regular and adaptable data foundation for subsequent distillation tasks.

[0055] Optionally, in one embodiment of this application, the attention graph sets of the teacher network layer and the student network layer in any distillation layer of the distillation layer set are obtained respectively, and the attention graph sets are grouped and integrated to obtain attention graph groups that meet the preset quantity requirements. The method further includes: determining whether the number of attention graphs in the attention graph group corresponding to the teacher network layer and the attention graph group corresponding to the student network layer are the same; if the number of attention graphs in the attention graph group corresponding to the teacher network layer and the attention graph group corresponding to the student network layer are the same, then constructing attention graph groups of the same quantity for both the teacher network layer and the student network layer; if the number of attention graphs in the attention graph group corresponding to the teacher network layer and the attention graph group corresponding to the student network layer are different, then adjusting the number of attention graphs in the last group of attention graphs corresponding to both the teacher network layer and the student network layer to obtain attention graph groups that meet the preset principle of equal number of attention graphs.

[0056] In the actual partitioning process, the embodiments of this application adhere to the principle of "making the number of attention maps in each group as equal as possible." If the total number of attention maps ( or Unable to be If divisible, the number of attention maps in the last group is allowed to be slightly less than the other groups (i.e., if the total number of attention maps in any distillation layer corresponding to the teacher network or student network cannot be divided by the preset number of groups). Divisibility is allowed (the number of attention maps in the last group can differ from the number of other groups by 1), in order to balance grouping fairness and operational feasibility.

[0057] For example, when teacher networks have Attention map, student network Each attention map is divided into... When grouping, the teacher network is grouped as [1,2,3], [4,5,6], and the student network is grouped as [1,2,3], [4,5,6]; when the teacher network... Student Network Divided into When grouping, the teacher network is grouped as [1,2,3], [4,5], [6,7], and the student network is grouped as [1,2], [3,4], [5].

[0058] Therefore, this application embodiment determines whether the number of attention graph groups in the teacher-student network layer is consistent to ensure the rationality of grouping, and adjusts the number of the last group when the number is different, thereby ensuring that the number of attention graphs in each group is equal, providing balanced data for subsequent distillation and improving the model distillation effect.

[0059] In step S103, the matrix dimensions corresponding to the teacher aggregation matrix and the student aggregation matrix are determined respectively, so as to construct the learnable transformation matrices corresponding to the teacher aggregation matrix and the student aggregation matrix according to the matrix dimensions. Then, matrix multiplication operations are performed on the teacher aggregation matrix and the student aggregation matrix with the corresponding learnable transformation matrices to obtain the corresponding teacher feature matrix and student feature matrix.

[0060] Based on the aforementioned grouping strategy of "averaging distribution according to the order of attention map generation numbers," this embodiment uses the corresponding attention map groups in the teacher network and student network (denoted as...) The range of values ​​for i is This paper will take [the example] as an example for detailed analysis. Since the processing methods for all attention map groups are completely consistent, this application embodiment focuses on [specific examples]. The analytical conclusions can be directly generalized to other groups.

[0061] Specifically, in the corresponding group In Chinese: Subgroups of the teacher network include An attention map (i.e., from the teacher network layer) Each attention map is assigned to The quantity is ); Subgroups of the student network contain An attention map (i.e., from the student network layer) Each attention map is assigned to The quantity is ).

[0062] It should be noted that, since the core function of attention graphs is to characterize the strength of associations among n tokens, with rows and columns corresponding to different tokens, the matrix dimensions of all attention graphs in the teacher and student networks are consistent, and each attention graph is a single matrix. A square matrix (n represents the number of tokens input to the current layer).

[0063] To integrate attention information within a group and perform subsequent feature alignment, embodiments of this application can... Attention maps within groups are stitched together: for teacher networks In this embodiment of the application, attention maps can be sequentially concatenated column by column (i.e., all columns of the first attention map are followed by all columns of the second attention map, and so on) to form a new matrix (i.e., a teacher aggregation matrix). Since each attention map is , After the images are concatenated column by column, the matrix is ​​formed. The dimension is ,Right now ;

[0064] Similarly, for student networks Each attention map is concatenated into a matrix (i.e., a student aggregation matrix) using the same column arrangement. Its dimensions are ,Right now .

[0065] Therefore, through the above-mentioned splicing operation, the information of multiple attention maps within a group is integrated into a matrix, laying a data foundation for subsequent solutions to dimensionality differences and the construction of distillation loss.

[0066] Those skilled in the art should understand that in knowledge distillation, the two objects involved in constructing the loss function must have the same dimension; otherwise, similarity comparison or difference measurement cannot be directly performed. In the intra-group alignment process of this application embodiment, this problem specifically manifests as a dimensional mismatch between the teacher aggregation matrix and the student aggregation matrix: the teacher aggregation matrix... The dimension is ( Input the number of tokens. (Number of attention maps within teacher groups); student aggregation matrix. The dimension is ( (Number of attention maps within student groups). Due to and They are usually not equal (because teachers and students have different numbers of heads to focus on). and number of columns and The differences between the two methods mean that they cannot be directly used to construct distillation losses.

[0067] To solve this problem, such as Figure 2 As shown, this application embodiment introduces a learnable matrix (i.e., the learnable transformation matrix corresponding to the teacher aggregation matrix, also known as the teacher learnable transformation matrix). Its dimensions are Through matrix multiplication ,Will Transformation OK, A matrix of columns (i.e., a square matrix of teacher characteristics). Similarly, embodiments of this application may also introduce a learnable matrix (i.e., the learnable transformation matrix corresponding to the student aggregation matrix, also known as the student learnable transformation matrix). Its dimensions are Through matrix multiplication ,Will Transformation OK, The column matrix (i.e., the student characteristic matrix) .

[0068] In actual implementation, the dimensions of the learnable transformation matrix are set as follows: the dimension of the learnable matrix that adapts to the teacher aggregation matrix is ​​(number of columns in the teacher aggregation matrix × side length of the target matrix), and the dimension of the learnable matrix that adapts to the student aggregation matrix is ​​(number of columns in the student aggregation matrix × side length of the target matrix).

[0069] Learnable matrix and The parameters are continuously optimized during model training. Their role is to learn the most representative features from the original aggregated matrix, achieving an effective mapping from high-dimensional concatenated information to a low-dimensional square matrix, rather than simple dimensionality compression. and In the case of equality, the embodiments of this application still employ a learnable matrix. Will Transform into Learnable matrices are used Will Transform into .

[0070] Therefore, in the embodiments of this application, for each group of teacher aggregation matrix and student aggregation matrix, a learnable transformation matrix adapted to their respective dimensions is introduced, and the two are transformed into teacher feature matrix and student feature matrix with consistent dimensions through matrix multiplication. This not only solves the problem of mismatched dimensions of teacher and student aggregation matrices, but also effectively improves the distillation effect.

[0071] As one possible approach, the specific process of constructing and optimizing the learnable matrix in this embodiment is described below:

[0072] Step 1: Construct a learnable matrix based on the dimensions n×(p×n) of the teacher aggregation matrix and n×(q×n) of the student aggregation matrix. and ,in, and It is initialized as a random matrix that follows an orthogonal distribution, and an attention head importance weight vector is introduced to weight the features of each attention head of the teacher and the student during matrix initialization, highlighting the feature distribution characteristics of key attention heads;

[0073] Step 2: Combine the teacher aggregation matrix with... Perform matrix multiplication to obtain the transformed teacher group matrix, i.e., the teacher characteristic matrix; then combine the student aggregation matrix with... Perform matrix multiplication to obtain the transformed student group matrix, i.e., the student feature matrix, and the learnable matrix output in step 1. and The weighted attention head features are used as the operands for matrix multiplication with the teacher aggregation matrix and the student aggregation matrix in this step, respectively.

[0074] Step 3: During model training, the backpropagation algorithm is used to apply distillation loss to... and The parameters are iteratively updated, and the matrix parameters of the learnable matrix are dynamically adjusted based on the activation frequency of the attention head, so that... and The most representative features can be learned from the original high-dimensional aggregation matrix. The teacher feature matrix and student feature matrix output in step 2 are used to calculate the distillation loss. This loss, along with the dynamically adjusted matrix parameters, serves as the gradient source for parameter updates in step 3.

[0075] Step 4: After each iteration update, and Orthogonality constraint regularization is performed by adding an orthogonal regularization loss term to the total loss, ensuring that the matrix maintains good feature preservation ability throughout the optimization process. This regularization loss, together with the distillation loss, guides the parameter update.

[0076] Therefore, the embodiments of this application can achieve more accurate capture of key features by using attention head weighting, dynamic weight adjustment and orthogonal regularization, thereby reducing information loss in high-dimensional to low-dimensional mapping and significantly improving the efficiency and effectiveness of knowledge distillation.

[0077] In step S104, based on the teacher feature matrix and the student feature matrix, the double distillation loss function corresponding to the natural language processing model is determined. Based on the double distillation loss function, the student network is trained, and the trained student network is used to infer the text information input by the target user and generate the corresponding response information.

[0078] Subsequently, embodiments of this application can also construct a dual distillation loss function covering fine-grained distribution and coarse-grained structure, and calculate group distillation loss based on teacher feature matrix and student feature matrix; secondly, embodiments of this application can also determine the distillation loss of the corresponding layer (i.e., the total loss of a single layer) based on the distillation loss of all groups, and then combine the distillation loss of all "one-to-one" distillation layers to obtain the total distillation loss (i.e., the distillation loss function), so as to guide student network training based on the total distillation loss.

[0079] Therefore, the embodiments of this application can be well applied to knowledge distillation operations in scenarios where the number of attention graphs is mismatched due to the difference in the number of attention heads between the teacher network and the student network in the Transformer architecture, thereby improving the distillation effect and the performance of the student network.

[0080] Optionally, in one embodiment of this application, determining the double distillation loss function corresponding to the natural language processing model based on the teacher feature matrix and the student feature matrix includes: normalizing the teacher feature matrix and the student feature matrix respectively to obtain the corresponding attention probability distribution matrix, and performing a downsampling operation on the teacher feature matrix and the student feature matrix to obtain the corresponding downsampling matrix; calculating the fine-grained distribution loss and the coarse-grained structure loss corresponding to the natural language processing model based on the attention probability distribution matrix and the downsampling matrix; and performing a weighted summation operation on the fine-grained distribution loss and the coarse-grained structure loss to obtain the double distillation loss function of the natural language processing model.

[0081] In the embodiments of this application, the double distillation loss function of the natural language processing model is constructed as follows:

[0082] 1. Normalize the teacher feature matrix and the student feature matrix respectively to obtain the attention probability distribution matrix;

[0083] 2. Simultaneously, downsampling is performed on the teacher feature matrix and the student feature matrix to obtain the corresponding downsampling matrix.

[0084] 3. Based on the attention probability distribution matrix and the downsampling matrix, calculate the fine-grained distribution loss and the coarse-grained structure loss of the natural language processing model, respectively.

[0085] 4. Weighted summation of the two losses yields the double-distillation loss function corresponding to the attention graph groups of the teacher and student network layers. .

[0086] Therefore, the embodiments of this application obtain two types of matrices through normalization and downsampling, and simultaneously calculate fine and coarse-grained losses, thereby enabling a more comprehensive constraint on the model and improving the performance and stability of the distilled student network.

[0087] Optionally, in one embodiment of this application, the fine-grained distribution loss and coarse-grained structure loss corresponding to the natural language processing model are calculated based on the attention probability distribution matrix and the downsampling matrix, including: normalizing the teacher feature matrix and the student feature matrix respectively to obtain the corresponding attention probability distribution matrix; comparing the elements in the attention probability distribution matrix with a preset probability value threshold to determine multiple target elements greater than the probability value threshold, and performing an element-by-element XOR operation on the multiple target elements to obtain a first operation result; performing probability decay on the elements in the attention probability distribution matrix other than the multiple target elements to obtain the corresponding decay probability value, and performing an element-by-element XOR operation on the decay probability value to obtain the corresponding second operation result; and determining the XOR operation result corresponding to the attention probability distribution matrix based on the first operation result and the second operation result, so as to calculate the fine-grained distribution loss based on the XOR operation result.

[0088] In the specific implementation process, the embodiments of this application can be applied to the teacher feature matrix after dimensional unification. and student characteristic matrix After performing Softmax normalization, a new matrix is ​​obtained (i.e., the attention probability distribution matrix of the teacher feature matrix, which can be called the teacher attention probability distribution matrix). And (i.e., the attention probability distribution matrix corresponding to the student feature matrix, which can be called the student attention probability distribution matrix) .

[0089] It's important to note that, functionally, the core function of the Softmax function is to perform probability normalization on the element values ​​of each row in the matrix. Through this operation, the value range of each element in the matrix is ​​constrained to [0,1], and the sum of all elements in each row is 1, ultimately forming a distribution vector in each row that conforms to probabilistic statistical laws. This transformation operation has clear technical significance: the core value of the attention map in the Transformer architecture is to characterize the correlation strength between input tokens (the larger the element value, the stronger the correlation between the two corresponding tokens). The Softmax operation can transform this "correlation strength" into "attention allocation probability." The normalized matrix elements can intuitively reflect the model's attention weight ratio to all other tokens when processing the current token, providing a feature carrier that conforms to cognitive logic for subsequent loss calculation based on probability distribution differences.

[0090] Subsequently, the process of calculating the fine-grained distribution loss based on the attention probability distribution matrix in this embodiment is as follows:

[0091] (1) Obtain the attention probability distribution matrix and set a probability threshold that is compatible with the feature scene. This threshold is dynamically adjusted according to the overall distribution of the matrix.

[0092] (2) Compare the matrix elements with the threshold element by element, mark multiple target elements that are greater than the threshold, and extract the position information and associated feature dimensions of these elements;

[0093] (3) Perform an element-by-element XOR operation on the target element, assign corresponding weights to the operation results based on the importance of its associated feature dimensions, and integrate them to obtain the first operation result;

[0094] (4) Based on the location information of the target element, locate the other elements in the attention probability distribution matrix except for the target element, and according to the distance between the element and the target element and its own probability value, adopt a step-by-step decay strategy to decay the probability. The greater the distance and the lower the initial probability of the element, the greater the decay amplitude, and obtain the decay probability value.

[0095] (5) Perform an element-wise XOR operation on the decay probability value, introduce the feature sparsity coefficient to adjust the operation result, and generate a second operation result;

[0096] (6) Analyze the distribution characteristics of the first and second operation results, and assign fusion weights to the first and second operation results according to the sensitivity of the feature scenario to key information. The first operation result has a higher weight in the key scenario.

[0097] (7) The weighted fusion of the two types of operation results is used to obtain the overall XOR operation result. The result is calibrated by combining the global distribution difference of the matrix dimension. Based on the calibrated XOR operation result, the first distillation loss, i.e. the fine-grained distribution loss, is calculated.

[0098] Furthermore, it is understood that in the embodiments of this application, the first distillation loss (i.e., the loss covering fine-grained distribution) is intended to be quantified. and The differences constrain the attention patterns of students' online learning and teachers' online learning, therefore, the first distillation loss The mathematical expression is:

[0099]

[0100] Among them, the function The loss function is used to measure the difference between two feature matrices (or distributions). Its core function is to maintain and quantify the deviation of the teacher network and student network in attention distribution, and to guide the student network to adjust parameters through backpropagation to reduce this deviation.

[0101] It should be noted that the embodiments of this application may specifically employ the following methods to construct functions. :

[0102] 1. Constructing functions using Kullback-Leibler divergence. It is used to quantify the differences between the feature maps of teacher networks and student networks in order to guide the learning process of student networks;

[0103] 2. Construct functions using mean squared error loss, cosine similarity, etc. .

[0104] When using Kullback-Leibler divergence, due to and for The matrix (each row corresponds to the attention distribution of a token) requires calculating the divergence of the probability distribution in each row and then taking the mean. The calculation expression is as follows:

[0105]

[0106] in, This is the attention distribution matrix of the teacher network (i.e., the teacher attention probability distribution matrix). ; This is the attention distribution matrix of the student network (i.e., the student attention probability distribution matrix). ; Let represent the attention probability of the i-th token to the j-th token in the teacher network; Let represent the attention probability at the corresponding position in the student network. The final KL divergence loss is obtained by averaging the divergence values ​​across all rows. At this point, the first distillation loss (i.e., the fine-grained distribution loss) is:

[0107]

[0108] Therefore, the embodiments of this application perform Softmax normalization operations on the teacher feature matrix and student feature matrix after unifying the dimensions to obtain the attention probability distribution matrix, and obtain the fine-grained distribution loss by quantifying the difference of the attention probability distribution matrix, thereby providing reliable data for loss calculation and accurately capturing the subtle differences in the attention distribution of teachers and students, which helps to optimize the student model.

[0109] As one possible approach, the specific operation process for quantifying the differences between the feature maps of the teacher network and the student network in this application embodiment is as follows:

[0110] Step 1: Perform dynamic temperature coefficient Softmax normalization on the teacher feature matrix and student feature matrix to obtain the teacher attention probability distribution matrix and student attention probability distribution matrix; the temperature coefficient is dynamically adjusted based on the entropy value of each row of the feature matrix, the higher the entropy value, the smaller the temperature coefficient; at the same time, the normalized matrix is ​​subjected to sparsity constraint processing, and low-weight elements are suppressed by L1 regularization to strengthen the probability ratio of key token pairs.

[0111] Step 2: Calculate the association strength weight of each token pair in the teacher and student attention probability distribution matrix. The association strength is obtained by the ratio of the product of the corresponding elements of the two matrices to the global mean. Introduce a cross-modal token type matching coefficient to correct the association strength weight.

[0112] Step 3: Use the association strength weight as the row weight for calculating KL divergence, and sum the KL divergence of each row to obtain the initial weighted KL divergence loss; then combine the topological similarity of the matrix row and column dimensions (obtained by calculating the cosine similarity of the row and column mean sequences) to make a secondary adjustment to the initial loss, and finally obtain the difference quantization function output.

[0113] It should be noted that the probability distribution matrix with sparsity constraints output in step 1 provides more focused basic data for step 2; the corrected association strength weights output in step 2 serve as the core input for step 3; and the final loss output in step 3 is used to guide the parameter optimization of the student network, while also feeding back to step 1 to dynamically adjust the entropy threshold of the temperature coefficient.

[0114] Therefore, the embodiments of this application can enhance feature focus through sparse constraints and type matching, and improve the comprehensiveness of loss quantification by combining topological similarity correction, thereby further optimizing the learning ability of student networks for cross-modal key features.

[0115] Optionally, in one embodiment of this application, calculating the fine-grained distribution loss and coarse-grained structure loss corresponding to the natural language processing model based on the attention probability distribution matrix and the downsampling matrix further includes: performing downsampling operations on the teacher feature matrix and the student feature matrix to obtain the corresponding downsampling matrices; calculating the average value of the downsampling matrices corresponding to the teacher feature matrix and the student feature matrix respectively, setting the elements in the downsampling matrices greater than the average value as a first target value, and setting the elements in the downsampling matrices less than or equal to the average value as a second target value, so as to determine the binarized sampling matrix of the downsampling matrix corresponding to the teacher feature matrix and the student feature matrix based on the first target value and the second target value; performing an element-wise XOR operation on the binarized sampling matrix corresponding to the teacher feature matrix and the student feature matrix to obtain the difference quantization matrix between the teacher feature matrix and the student feature matrix, and performing average pooling processing on the difference quantization matrix to obtain the coarse-grained structure loss.

[0116] Furthermore, embodiments of this application can also calculate a second distillation loss (i.e., a loss covering coarse-grained structures) to capture the global structural consistency of the attention map, supplementing the first distillation loss's focus on local numerical differences, such as... Figure 2 As shown, the calculation process for the second distillation loss is as follows:

[0117] 1. MaxPool downsampling (i.e., maximum pooling sampling to extract locally strongly correlated features and reduce complexity):

[0118] For the teacher feature matrix obtained after dimensional unification (an n×n square matrix, where n is the number of input tokens). First, apply the MaxPool operator of size 3×3 with a step size of 3 to obtain the downsampled matrix (i.e., the downsampled matrix corresponding to the teacher feature matrix, also known as the teacher downsampled matrix). The core technical logic and parameter selection criteria for this operation are as follows:

[0119] In the attention graph, adjacent elements usually correspond to local associations between tokens (such as semantic associations between adjacent words in text, spatial associations between adjacent pixels in an image). The MaxPool operator can filter the maximum attention value within each local window, retain the teacher network's judgment on "local key associations", and at the same time remove redundant weak association information to achieve feature purification.

[0120] A 3×3 pooling kernel can cover the local association range of three surrounding tokens of a single token, avoiding the lack of downsampling effect of 1×1 pooling (inability to reduce dimensionality) and preventing excessive blurring of local details by 5×5 pooling (loss of key association location information); setting both the stride and the pooling kernel size to 3 can achieve non-overlapping downsampling; if n is an integer multiple of 3 (e.g., n=768). The dimension will be compressed to (n / 3)×(n / 3) (e.g., 256×256), reducing the number of matrix elements to 1 / 9 of the original, significantly reducing subsequent computational overhead; if n is not an integer multiple of 3 (e.g., n=512), it is necessary to first... Perform zero padding at the edges to ensure that the dimensions meet the specified requirements after padding. ( (To round up), ensure that the pooling output dimension is an integer, avoiding dimension mismatch.

[0121] Student feature matrix after unifying student-side dimensions , adopt and Using identical MaxPool parameters and operating procedures, we obtain the downsampling matrix (i.e., the downsampling matrix corresponding to the student feature matrix, also known as the student downsampling matrix). This ensures that the feature extraction standards are consistent between the teacher and student sides.

[0122] 2. Thresholding (abstracting attention features into a binary structure):

[0123] (1) To Thresholding is performed to generate a binary matrix (i.e., the binarized sampling matrix corresponding to the downsampling matrix of the teacher feature matrix, also known as the teacher binarized sampling matrix). :

[0124] 1) Mean calculation:

[0125] First calculate The average of all elements in the matrix, the average represents The average attention intensity can be used as a threshold for dividing "key attention areas" and "non-key areas". Elements above the mean value correspond to strongly associated areas as determined by the teacher network, while elements below the mean value correspond to weakly associated or unassociated areas.

[0126] 2) Binarization transformation (based on average value) (Thresholding)

[0127] Elements greater than or equal to the average are set to 1 (marked as key areas of interest), and elements less than the average are set to 0 (marked as non-key areas), resulting in a binary matrix containing only 0s and 1s. The core purpose of this operation is "structural abstraction," which strips away the specific attention intensity values ​​and retains only the binary structural information of "whether attention is received," laying the foundation for subsequent loss calculations focusing on structural consistency.

[0128] 3) Synchronous processing on the student side:

[0129] right Perform the same process (calculate the average, generate thresholds) (That is, the binarized sampling matrix corresponding to the downsampling matrix of the student feature matrix, also known as the student binarized sampling matrix), and the binarization rules are completely consistent with those of the teacher side, ensuring the uniformity of the abstract logic of the structure on both sides and avoiding the introduction of additional errors due to differences in processing standards.

[0130] 3. XOR operation (quantifies structural differences and calculates single-group loss):

[0131] The second distillation loss can be addressed by... and The result is obtained by performing an element-by-element XOR operation, as follows:

[0132] (1) XOR operation rules:

[0133] The logic of the XOR operation is: "If the two elements have the same value, the result is 0; if the values ​​are different, the result is 1." and If both are 1 (both teachers and students are judged as critical areas) or both are 0 (both are judged as non-critical areas), the result is 0 (indicating structural consistency); if one is 1 and the other is 0 (judgment results conflict), the result is 1 (indicating structural difference).

[0134] (2) Loss calculation:

[0135] Perform average pooling on the result matrix after the XOR operation to obtain a single group. Second distillation loss As shown in the following formula:

[0136] Where k = n / 3; the range of the second distillation loss value is [0,1]. The larger the value, the more significant the difference in the macroscopic structure of attention between teachers and students; the smaller the value, the stronger the consistency of the coarse-grained structure of attention.

[0137] (3) Complementarity with the first distillation loss:

[0138] The first distillation loss (such as KL divergence) focuses on the "fine numerical differences" in attention distribution (e.g., the teacher network's attention weight for a certain token is 0.8, while the student's is 0.6), while the second distillation loss focuses on the "macrostructural differences" (e.g., whether the teacher and student identify the same area as a key area of ​​attention). Combining the two creates a dual constraint; for example, if the first distillation loss is small but the second distillation loss is large, it indicates that although the student network imitates the teacher network's local attention values, its overall area of ​​attention deviates. In this case, the second distillation loss can promptly correct this problem, ensuring the comprehensiveness of knowledge transfer.

[0139] Therefore, the embodiments of this application use MaxPool downsampling, thresholding, and XOR operation to calculate the coarse-grained structural loss, thereby capturing the global structural consistency of the teacher and student attention map. This complements the first distillation loss that focuses on local numerical differences, forming a dual constraint to effectively ensure the comprehensiveness of knowledge transfer.

[0140] Optionally, in one embodiment of this application, a double distillation loss function corresponding to the natural language processing model is determined based on the teacher feature matrix and the student feature matrix, and the student network is trained based on the double distillation loss function, including: calculating the single-layer total loss of the distillation layer corresponding to the attention map group based on the double distillation loss function; calculating the mean of the single-layer total loss corresponding to the set of distillation layers based on the single-layer total loss, and determining the distillation loss function corresponding to the natural language processing model based on the mean of the single-layer total loss; obtaining the input word sequence of the natural language processing model, and determining the probability distribution vector of words at different positions in the input word sequence belonging to different words in a preset word table, and determining the cross-entropy loss function corresponding to the natural language processing model based on the probability distribution vector; constructing the total loss function corresponding to the natural language processing model based on the distillation loss function and the cross-entropy loss function, and training the student network using the total loss function.

[0141] It should be noted that, due to the limitations of a single loss function in capturing the knowledge differences between teacher and student networks and its inability to comprehensively cover knowledge features across different dimensions, the embodiments of this application may use the first distillation loss. With the second distillation loss To achieve more comprehensive knowledge alignment, attention graphs are integrated. The total loss (i.e., the double distillation loss function) is calculated using the following formula:

[0142]

[0143] in, It is a constant, and can generally be set to 1.0.

[0144] Therefore, this embodiment of the application achieves full-dimensional knowledge complementarity and alignment between fine-grained attention probability distribution and coarse-grained attention structural features by directly summing the first distillation loss and the second distillation loss, without the need for additional parameter tuning. At the same time, the dual loss signals work together to avoid knowledge replication bias dominated by a single loss, thus balancing distillation effect, training stability and efficiency.

[0145] Secondly, for the selected "one-to-one" distillation layer mentioned above (This layer contains a total of) (corresponding groups), their single-layer total loss The average loss for all groups in this layer is calculated using the following formula:

[0146]

[0147] In the embodiments of this application, within the same layer Each group corresponds to a different subset of the attention map, and the knowledge information carried by each group differs to some extent. This embodiment of the application uses an averaging operation to avoid the extreme loss values ​​of a particular group from excessively affecting the overall layer loss, thereby effectively balancing the knowledge contributions of each group to more stably and objectively reflect the knowledge alignment effect of the layer, providing a reliable basis for subsequent calculation of the overall distillation loss.

[0148] Subsequently, embodiments of this application can calculate all selected... The average value of the loss of each "one-to-one" distillation layer (i.e., the total loss of a single layer) is used to obtain the average total loss of a single layer, and the distillation loss function of the entire knowledge distillation process is determined based on this average total loss of a single layer.

[0149] Specifically, embodiments of this application can improve performance by averaging the knowledge of different network layers (such as basic features at lower levels and semantic features at higher levels) across all participating network layers. This allows the student network to learn from the multi-level knowledge of the teacher network. The mathematical expression for this distillation loss function is as follows:

[0150]

[0151] Finally, in this embodiment, the total loss function corresponding to the natural language processing model can be constructed based on the distillation loss function and the pre-built cross-entropy loss function.

[0152] Therefore, the embodiments of this application achieve full-dimensional knowledge alignment by fusing double distillation loss (without additional parameter tuning), and can balance the knowledge contributions of each network layer, avoid extreme effects, and integrate cross-entropy loss, thereby effectively improving the performance, training stability and efficiency of the student network.

[0153] Optionally, in one embodiment of this application, the input word sequence of the natural language processing model is obtained, and the probability distribution vectors of words at different positions in the input word sequence belonging to different words in a preset word table are determined. Based on the probability distribution vectors, the cross-entropy loss function corresponding to the natural language processing model is determined, including: obtaining the probability distribution vectors of the next word at the current position in the input word sequence belonging to different words in the word table through the natural language processing model, and calculating the logarithm of the probability distribution vectors; determining the actual word category corresponding to the current position in the input word sequence, so as to obtain the cross-entropy loss function based on the actual word category and the logarithm.

[0154] In practical implementation, the embodiments of this application can perform knowledge distillation on natural language processing models (such as large language models) to predict the next word given a sequence of tokens. Therefore, the task loss function in the embodiments of this application... Cross-entropy loss function can be used.

[0155] Specifically, in the embodiments of this application, it can be assumed that there exists a component containing A sequence of n words, for each position Natural language processing models need to predict which word belongs to the vocabulary. The probability distribution of n words. Let... For one A one-hot vector of dimension, representing position. The category corresponding to the real word element at that location, Indicates the model prediction If the probability distribution vector is dimensional, then the cross-entropy loss function can be expressed as:

[0156]

[0157] in, express The first vector One element; express The first vector Each element.

[0158] Therefore, this embodiment calculates the average difference between the probability distribution predicted by the model and the true label using the cross-entropy loss function, and adjusts the model parameters by minimizing the loss function, thereby enabling the model to better predict the next word and effectively improving the model's performance.

[0159] Optionally, in one embodiment of this application, constructing a total loss function corresponding to the natural language processing model based on the distillation loss function and the cross-entropy loss function includes: determining the model processing requirements corresponding to the natural language processing model, and calculating the weight coefficients corresponding to the distillation loss function according to the model processing requirements; and performing linear operations on the distillation loss function and the cross-entropy loss function based on the weight coefficients to obtain the total loss function.

[0160] As one possible approach, the process of constructing the total loss function of the natural language processing model based on the distillation loss function and the cross-entropy loss function in this embodiment of the application is as follows:

[0161] 1. Weighting coefficient calculation stage:

[0162] In the embodiments of this application, the processing requirements of the natural language processing model directly determine the importance ratio of the two types of loss functions. For example, in text generation tasks with high accuracy requirements, more emphasis should be placed on the distillation loss function (to ensure that the student model fits the knowledge transfer of the teacher model); while in classification tasks with low resources, priority should be given to the cross-entropy loss function (to ensure the model's fit to the task labels). Based on these requirements, technicians will calculate the weight coefficients corresponding to the distillation loss function through experimental verification or empirical threshold setting. These coefficients must accurately match the task's need for a balance between "knowledge transfer accuracy" and "task fit".

[0163] 2. Total Loss Function Construction Stage:

[0164] After determining the weighting coefficients, embodiments of this application can use linear operations to fuse the two types of loss functions. Specifically, embodiments of this application can define a total loss function based on the distillation loss function and the cross-entropy loss function. As shown in the following formula:

[0165] .

[0166] It should be noted that, Based on the distillation loss function Task-related loss function (i.e., cross-entropy loss function) It is a function of the independent variable. In the embodiments of this application, A linear function is used, as shown in the following equation:

[0167]

[0168] in, Indicates constant coefficients used for balance. and The contribution, in general, It can be set to 0.5. For different tasks, the embodiments of this application can be adjusted. To achieve better model prediction accuracy.

[0169] Therefore, the embodiments of this application can determine the distillation loss weights in combination with the model processing requirements, and fuse the two types of losses through a linear operation function, and the coefficients can be adjusted as needed, thereby adapting to different task requirements, balancing knowledge transfer and task fitting, and helping the model achieve optimal accuracy in various scenarios.

[0170] Optionally, in one embodiment of this application, a student network is trained based on a double distillation loss function to infer text information input by a target user and generate corresponding response information. This includes: sampling a preset text database to obtain multiple text sampling data, preprocessing the multiple text sampling data to obtain corresponding multiple text sampling standard data, and constructing a training dataset for a natural language processing model using the multiple text sampling standard data; training a teacher network for the natural language processing model using the training dataset to obtain a trained teacher network, determining the network architecture of the student network based on the trained teacher network, and initializing the weight matrix and model parameters of the student network to obtain the initial weight matrix and model parameters corresponding to the student network. Initial parameters are defined; based on the training dataset, and combined with the initial weight matrix, initial model parameters, and network architecture, the student network is trained iteratively in multiple rounds. After each round of training, the order of multiple text sampling standard data in the training dataset is changed according to a preset transformation strategy to construct a corresponding scrambled training dataset. In the next round of training, the student network is trained in multiple batches using the scrambled training dataset. In each batch of training, the total loss function is calculated to obtain the corresponding total loss value. Training stops when the total loss value meets the preset convergence requirement, resulting in the trained student network. The text information input by the target user is obtained and input into the trained student network. The trained student network is then used to infer the text information to generate the corresponding response information.

[0171] It should be noted that the embodiments of this application can input text data into a natural language processing model to generate intermediate state values ​​during model operation (e.g., query embedding, key embedding, and value embedding), as well as the final output text (i.e., response information).

[0172] Specifically, in this embodiment, a large-scale text dataset containing hundreds of millions of sentences can be collected first, with Chinese sentences accounting for 30%, English sentences accounting for 60%, and other languages ​​accounting for 10%. During model training, this embodiment can use random sampling to sample text data from the large-scale text dataset to obtain multiple text sample data. These multiple text sample data are then preprocessed to obtain corresponding multiple text sample standard data. Finally, the training dataset for the natural language processing model is constructed using these multiple text sample standard data.

[0173] Secondly, to avoid the order of samples in the training dataset D affecting model performance, all samples in the training dataset D are randomly arranged at the beginning of model training. During each round of training, the model training algorithm reads a batch of samples, for example, 1024 samples.

[0174] Furthermore, embodiments of this application can read data from the training dataset D and input it into the teacher network (i.e., the source model) to obtain the trained teacher network. Based on the trained teacher network, the network architecture of the student network is determined, and the weight matrix and model parameters of the student network are initialized to obtain the initial weight matrix and initial model parameters corresponding to the student network. Specifically, embodiments of this application can initialize the training period. Total training cycle The stochastic gradient descent algorithm is the Adam algorithm; the learning rate is... momentum coefficient , , The learning rate strategy adopts the cosine strategy, and the specific adjustment method is to dynamically adjust the learning rate according to the cosine function during the training period; the number of samples in a mini-batch is B = 1024.

[0175] It should be noted that, in the embodiments of this application, the student network (i.e., the target model) The structure is basically the same as the source model, except for the number of heads in the multi-head attention mechanism. This difference exists in each layer of the target model. The number of heads in the multi-head attention mechanism is no greater than the number of heads in the source model's multi-head attention mechanism.

[0176] Subsequently, embodiments of this application may set the total loss function of the natural language processing model. And the target model of the natural language processing model. The training process is as follows:

[0177] 1. Source model All parameter values ​​are fixed; based on the source model Network structure construction target model Network structure; random initialization of the target model The element values ​​of the weight matrix, and other parameters;

[0178] 2. When When this happens, perform the following operations:

[0179] (1) ;

[0180] (2) The training dataset The order of the samples in the sample is randomly shuffled;

[0181] (3) From the training dataset Select a training sample from batch B;

[0182] (4) Calculate the total loss function based on the above training samples and parameter settings. The loss value is calculated, and the target model is updated using the stochastic gradient descent algorithm (Adam algorithm). The parameters are set; before updating the parameters, the gradient is clipped, and the gradient threshold is set to 5 (which can be adjusted according to the actual situation) to prevent gradient explosion.

[0183] (5) Repeat (3) and (4) until the training dataset is complete. All samples have been utilized, and the current training cycle ends.

[0184] 3. End training and output the target model (i.e., the trained student network). and the target model Save it to the specified file path for later use.

[0185] Therefore, the embodiments of this application can improve the efficiency and performance of student network training by constructing a large-scale multilingual dataset, setting a reasonable training strategy, and combining it with the constructed total loss function to train the teacher network and the student network.

[0186] Optionally, in one embodiment of this application, after using the trained student network to infer the text information input by the target user and generate the corresponding response information, the method further includes: sending the response information to the user terminal, and after the target user receives the response information, obtaining the multimodal interaction feedback data corresponding to the target user; performing multimodal fusion processing on the multimodal interaction feedback data to construct the corresponding dynamic feedback weight coefficients, and adjusting the weight coefficients of the distillation loss function based on the dynamic feedback weight coefficients, so as to re-weight and sum the distillation loss function and the cross-entropy loss function according to the adjusted weight coefficients of the distillation loss function, so as to construct the adaptive optimization total loss function corresponding to the natural language processing model; obtaining the historical interaction data corresponding to the natural language processing model, and constructing the optimized training samples corresponding to the natural language processing model based on the historical interaction data; inputting the optimized training samples into the natural language processing model to retrain the student network until the adaptive optimization total loss function meets the preset convergence requirement, so as to obtain the natural language processing optimization model; collecting new text information input by the target user, and inputting the new text information into the natural language processing optimization model to output the corresponding new response information.

[0187] In the specific implementation process, after the trained student network completes the initial inference (outputting the target user's text information response), the embodiments of this application can also achieve continuous improvement in model performance through multiple rounds of feedback and iterative optimization, as shown in the following process:

[0188] 1. Multimodal interactive feedback data acquisition:

[0189] This application embodiment can push the reply information generated by the student network to the user terminal (such as a web page). After the target user receives and views the reply, its multimodal interactive feedback data is collected. This multimodal interactive feedback data covers a variety of user interaction behaviors, such as text feedback (textual evaluation of the reply, such as "the answer is inaccurate" or "the logic is clear"), operation feedback (whether to click the "satisfied" or "unsatisfied" button, whether to ask a follow-up question), and even emotional feedback (if the device supports it, the user's facial expression changes, voice tone and emotion can be collected), to ensure that the user's true evaluation of the reply quality is fully captured.

[0190] 2. Construction of dynamic feedback weight coefficients and optimization of total loss function:

[0191] The collected multimodal interactive feedback data undergoes multimodal fusion processing. Specifically, in this embodiment, different types of feedback data are first standardized (e.g., text evaluations are converted into quantitative scores, and operational feedback is mapped to 0-1 values). Then, through an attention mechanism or weighted fusion algorithm, multi-dimensional feedback information is integrated to construct a dynamic feedback weight coefficient. This coefficient directly reflects the user's attention to the "model knowledge transfer effect" (corresponding to distillation loss) and the "task fitting accuracy" (corresponding to cross-entropy loss). For example, if the user frequently reports "response logic deviates from requirements" (due to insufficient knowledge transfer), the dynamic feedback weight coefficient will tilt towards distillation loss.

[0192] Secondly, the embodiments of this application can use the dynamic coefficient to adjust the weight of the distillation loss function, and re-weight and sum the distillation loss function and the cross-entropy loss function to obtain an adaptively optimized total loss function, so that the loss function can match user needs and feedback in real time.

[0193] 3. Optimize training sample construction and student network retraining:

[0194] This application embodiment can select highly relevant interaction data from the historical interaction data of the natural language processing model (including past user input text, model responses, user feedback records, etc.) as the basis for constructing optimized training samples; for example, if the user feedback is "low accuracy of translation-related responses", then interaction data related to historical translation tasks will be selected first as samples to ensure the relevance of the training samples.

[0195] Subsequently, in this embodiment, the optimized training samples can be input into the original natural language processing model, and the student network can be retrained with the adaptive optimization of the total loss function as the optimization objective. During the training process, the change of the loss function value is continuously monitored until it drops to a preset convergence threshold (e.g., the loss value is stable below 0.01 and there is no significant fluctuation for 5 consecutive rounds), then the training is stopped and the optimized natural language processing model is obtained.

[0196] 4. Optimize the inference application of the model:

[0197] When a target user inputs new text information (such as a new round of consultation or translation requests), the new text information is input into the optimized model. Based on the iteratively optimized parameters and knowledge, the model will generate new response information that better meets the user's needs, thus achieving a closed loop of "feedback-optimization-improvement".

[0198] Therefore, the embodiments of this application construct dynamic weights by collecting multimodal user feedback to optimize the total loss function, and at the same time optimize the sample retraining model in a targeted manner, thereby forming a closed-loop iteration, which enables the model to continuously meet user needs and greatly improves the model response quality.

[0199] Optionally, in one embodiment of this application, the process of constructing dynamic feedback weight coefficients and optimizing the total loss function is as follows:

[0200] Step 1: Collect multimodal interactive feedback data of the target user corresponding to the student's network output response after training. This data should include at least text evaluation feedback (such as "logical confusion", "information missing"), operation behavior feedback (such as "satisfied" / "unsatisfied" clicks, response dwell time, and frequency of follow-up questions), and emotional feature feedback (such as the tone intensity of voice feedback and facial expressions of image feedback).

[0201] Step 2: Classify and standardize the multimodal interaction feedback data collected in Step 1: For text evaluation feedback, a pre-trained text sentiment classification model is used to extract semantic tendencies and map them to a quantitative score of 0-10 (the higher the score, the more positive the feedback); for operation behavior feedback, it is converted into numerical values ​​according to preset rules (e.g., "Satisfied" click = 1, "Dissatisfied" click = 0, dwell time > 30 seconds = 0.8, 10-30 seconds = 0.5, < 10 seconds = 0.2, follow-up question = 0.3, no follow-up question = 0); for emotional feature feedback, an emotional intensity value is extracted through a voice emotion recognition model (values ​​range from 0-1, 1 represents the strongest emotion), and a facial expression recognition model outputs the corresponding score for the emotion category (e.g., "Pleasant" = 0.9, "Confused" = 0.3, "Dissatisfied" = 0.1), outputting the standardized multimodal feedback subset.

[0202] Step 3: Construct a feedback type importance evaluation module. First, based on the correlation analysis between historical feedback data and model performance, determine the initial weights of each feedback type (e.g., text evaluation weight 0.4, operational behavior weight 0.3, sentiment feature weight 0.3). Then, use an attention fusion algorithm to dynamically adjust the weight allocation according to the information density of the current feedback data (e.g., semantic richness of text evaluation, explicitness of operational behavior). Integrate the information of the standardized multimodal feedback subset output in Step 2 to calculate the dynamic feedback weight coefficient, which includes the proportion of distillation loss weight and the proportion of cross-entropy loss weight.

[0203] Step 4: Using the dynamic feedback weight coefficients output in Step 3 as the adjustment basis, update the weights of the distillation loss function and the cross-entropy loss function of the natural language processing model, and output the adaptively optimized total loss function. In this way, the loss function is dynamically adjusted through the fine-grained fusion of multimodal feedback, so that the model training direction matches the user's real needs.

[0204] Therefore, the embodiments of this application achieve accurate quantification of multimodal feedback through classification standardization, and dynamically adjust the feedback weight by combining attention fusion, thereby better aligning with the user's real evaluation logic, effectively improving the quality of responses and user satisfaction, while enhancing the flexibility and adaptability of model iteration.

[0205] It is understood that the embodiments of this application, through an integrated solution covering both fine-grained distribution and coarse-grained structure double distillation loss construction, attention grouping integration, unified learnable matrix dimensions, and double distillation loss constraints, effectively solve the problems of mismatched attention graph numbers and inconsistent dimensions caused by differences in the number of attention heads between teacher and student networks in traditional knowledge distillation, thus avoiding the loss of key information at the attention level during knowledge transfer. Simultaneously, the embodiments of this application can accurately capture the full-dimensional core knowledge (fine-grained distribution differences and coarse-grained structural features) at the attention level under the Transformer architecture, significantly improving the efficiency and effectiveness of knowledge distillation. This helps student networks fully absorb the attention patterns of teacher networks, enhancing the performance of student networks in tasks such as text processing and image recognition. Furthermore, the integrated solution of the embodiments of this application can be specifically adapted to the characteristics of the Transformer architecture, further expanding the adaptability and application value of knowledge distillation methods in Transformer-based neural network models, providing strong support for the efficient compression and performance optimization of Transformer-like models.

[0206] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0207] Embodiments of this application also provide a text message response device.

[0208] like Figure 3 As shown, the text message response device 10 includes: a network layer matching module 100, a group integration module 200, a matrix transformation module 300, and a text response module 400.

[0209] The network layer matching module 100 is used to select a preset number of teacher network layers that meet the preset network layer contribution requirements from the teacher network of the preset natural language processing model, and to select a preset number of student network layers that meet the preset network layer contribution requirements from the student network of the natural language processing model, and to perform network layer matching on the teacher network layers and student network layers to construct the corresponding distillation layer set.

[0210] The grouping and integration module 200 is used to obtain the attention map sets of the teacher network layer and the student network layer in any distillation layer of the distillation layer set, and to group and integrate the attention map sets to obtain attention map groups that meet the preset quantity requirements. The attention maps in the attention map groups are then spliced ​​column by column to generate the corresponding teacher aggregation matrix and student aggregation matrix.

[0211] The matrix transformation module 300 is used to determine the matrix dimensions corresponding to the teacher aggregation matrix and the student aggregation matrix respectively, so as to construct the learnable transformation matrices corresponding to the teacher aggregation matrix and the student aggregation matrix according to the matrix dimensions, and to perform matrix multiplication operations on the teacher aggregation matrix and the student aggregation matrix with the corresponding learnable transformation matrices respectively to obtain the corresponding teacher feature matrix and student feature matrix.

[0212] The text response module 400 is used to determine the double distillation loss function corresponding to the natural language processing model based on the teacher feature matrix and the student feature matrix. Based on the double distillation loss function, the student network is trained, and the trained student network is used to infer the text information input by the target user and generate the response information corresponding to the text information.

[0213] Optionally, in one embodiment of this application, the grouping integration module 200 includes: an attention map generation unit and a segmentation unit.

[0214] The attention map generation unit is used to generate multiple corresponding teacher-level attention maps based on the multi-head attention mechanism in the teacher network layer, and to generate multiple corresponding student-level attention maps using the multi-head attention mechanism in the student network layer.

[0215] The division unit is used to determine the generation sequence number of multiple teacher-level attention maps and multiple student-level attention maps respectively, and to divide the multiple teacher-level attention maps and multiple student-level attention maps into the corresponding attention map groups according to the generation sequence number, so as to obtain attention map groups that meet the preset quantity requirements.

[0216] Optionally, in one embodiment of this application, the text response module 400 includes: a normalization unit, a first loss calculation unit, and a weighted summation unit.

[0217] The normalization unit is used to normalize the teacher feature matrix and the student feature matrix respectively to obtain the corresponding attention probability distribution matrix, and to perform downsampling operation on the teacher feature matrix and the student feature matrix to obtain the corresponding downsampling matrix.

[0218] The first loss calculation unit is used to calculate the fine-grained distribution loss and coarse-grained structural loss of the natural language processing model based on the attention probability distribution matrix and the downsampling matrix.

[0219] The weighted summation unit is used to perform a weighted summation operation on the fine-grained distribution loss and the coarse-grained structure loss of the coverage to obtain the double distillation loss function of the natural language processing model.

[0220] Optionally, in one embodiment of this application, the text response module 400 further includes: a second loss calculation unit, a determination unit, a cross-entropy loss calculation unit, and a construction unit.

[0221] The second loss calculation unit is used to calculate the total single-layer loss of the distillation layer corresponding to the attention map group based on the double distillation loss function.

[0222] The unit is defined to calculate the mean of the total loss of a single layer corresponding to the set of distillation layers based on the total loss of a single layer, and to determine the distillation loss function corresponding to the natural language processing model based on the mean of the total loss of a single layer.

[0223] The cross-entropy loss calculation unit is used to obtain the input word sequence of the natural language processing model and determine the probability distribution vector of words at different positions in the input word sequence belonging to different words in the preset word table, so as to determine the cross-entropy loss function corresponding to the natural language processing model based on the probability distribution vector.

[0224] The building unit is used to construct the total loss function corresponding to the natural language processing model based on the distillation loss function and the cross-entropy loss function, and to train the student network using the total loss function.

[0225] Optionally, in one embodiment of this application, the text response module 400 further includes: a sampling unit, a first training unit, a second training unit, and an inference unit.

[0226] The sampling unit is used to sample a preset text database to obtain multiple text sampling data, and to preprocess the multiple text sampling data to obtain multiple corresponding text sampling standard data. The training dataset of the natural language processing model is constructed through the multiple text sampling standard data.

[0227] The first training unit is used to train the teacher network of the natural language processing model using the training dataset to obtain the trained teacher network. Based on the trained teacher network, the network architecture of the student network is determined, and the weight matrix and model parameters of the student network are initialized to obtain the initial weight matrix and initial model parameters of the student network.

[0228] The second training unit is used to perform multiple rounds of iterative training on the student network based on the training dataset and in combination with the initial weight matrix, initial model parameters, and network architecture. After each round of training, the order of multiple text sampling standard data in the training dataset is changed according to a preset transformation strategy to construct a corresponding scrambled training dataset. In the next round of training, the student network is trained in multiple batches using the scrambled training dataset. In each batch of training, the total loss function is calculated to obtain the corresponding total loss value. Training stops when the total loss value meets the preset convergence requirements to obtain the trained student network.

[0229] The inference unit is used to acquire text information input by the target user, input the text information into the trained student network, and use the trained student network to infer the text information to generate the corresponding response information.

[0230] Optionally, in one embodiment of this application, the first loss calculation unit includes: a normalization calculation subunit, a comparison subunit, a probability decay subunit, and an XOR subunit.

[0231] The normalization calculation subunit is used to perform normalization calculations on the teacher feature matrix and the student feature matrix respectively to obtain the corresponding attention probability distribution matrix.

[0232] The comparison sub-unit is used to compare the elements in the attention probability distribution matrix with a preset probability value threshold to identify multiple target elements that are greater than the probability value threshold, and to perform an element-by-element XOR operation on the multiple target elements to obtain the first operation result.

[0233] The probability decay subunit is used to perform probability decay on elements in the attention probability distribution matrix other than multiple target elements to obtain the corresponding decay probability value, and to perform an element-by-element XOR operation on the decay probability value to obtain the corresponding second operation result.

[0234] The XOR subunit is used to determine the XOR operation result corresponding to the attention probability distribution matrix based on the first operation result and the second operation result, so as to calculate the coverage fine-grained distribution loss according to the XOR operation result.

[0235] Optionally, in one embodiment of this application, the first loss calculation unit further includes: a downsampling subunit, a mean calculation subunit, and a difference quantization unit.

[0236] The downsampling subunit is used to perform downsampling operations on the teacher feature matrix and the student feature matrix to obtain the corresponding downsampling matrix.

[0237] The mean calculation subunit is used to calculate the average value of the downsampling matrices corresponding to the teacher feature matrix and the student feature matrix, respectively. Elements in the downsampling matrix that are greater than the average value are set as the first target value, and elements in the downsampling matrix that are less than or equal to the average value are set as the second target value. Based on the first target value and the second target value, the binarized sampling matrix of the downsampling matrix corresponding to the teacher feature matrix and the student feature matrix is ​​determined.

[0238] The differential quantization unit is used to perform element-wise XOR operations on the binary sampling matrices corresponding to the teacher feature matrix and the student feature matrix to obtain the differential quantization matrix between the teacher feature matrix and the student feature matrix, and then perform average pooling on the differential quantization matrix to obtain the coarse-grained structure loss.

[0239] Optionally, in one embodiment of this application, the cross-entropy loss calculation unit includes: a logarithmic calculation subunit and a category determination subunit.

[0240] The logarithm calculation subunit is used to obtain the probability distribution vector of the next word in the current position of the input word sequence belonging to different words in the word table through the natural language processing model, and to calculate the logarithm of the probability distribution vector.

[0241] The category determination subunit is used to determine the actual word category corresponding to the current position in the input word sequence, so as to obtain the cross-entropy loss function based on the actual word category and the logarithm.

[0242] Optionally, in one embodiment of this application, the construction unit includes: a weight coefficient calculation subunit and a linear calculation subunit.

[0243] The weight coefficient calculation subunit is used to determine the model processing requirements corresponding to the natural language processing model, and to calculate the weight coefficients corresponding to the distillation loss function based on the model processing requirements.

[0244] The linear computation subunit is used to perform linear operations on the distillation loss function and the cross-entropy loss function based on the weighting coefficients to obtain the total loss function.

[0245] Optionally, in one embodiment of this application, the grouping integration module 200 further includes: a judgment unit, a first analysis unit, and a second analysis unit.

[0246] The judgment unit is used to determine whether the number of attention maps in the attention map group corresponding to the teacher's network layer and the attention map group corresponding to the student's network layer are the same.

[0247] The first analysis unit is used to construct the same number of attention graph groups for the teacher network layer and the student network layer if the number of attention graph groups corresponding to the teacher network layer and the student network layer are the same.

[0248] The second analysis unit is used to adjust the number of attention maps in the last attention map group corresponding to the teacher network layer and the student network layer if the number of attention maps in the attention map group corresponding to the teacher network layer and the attention map group corresponding to the student network layer are not the same, so as to obtain an attention map group that satisfies the preset principle of equal number of attention maps.

[0249] Optionally, in one embodiment of this application, the text message response device 10 further includes: a receiving module, a multimodal fusion module, a selection module, an optimization module, and an updating module.

[0250] The receiving module is used to infer the text information input by the target user using the trained student network, generate the corresponding response information, send the response information to the user terminal, and obtain the multimodal interaction feedback data corresponding to the target user after the target user receives the response information.

[0251] The multimodal fusion module is used to perform multimodal fusion processing on multimodal interactive feedback data to construct corresponding dynamic feedback weight coefficients. Based on the dynamic feedback weight coefficients, the weight coefficients of the distillation loss function are adjusted. The distillation loss function and cross-entropy loss function are then re-weighted and summed according to the adjusted weight coefficients to construct the adaptive optimization total loss function corresponding to the natural language processing model.

[0252] The selection module is used to obtain historical interaction data corresponding to the natural language processing model, and to construct optimized training samples corresponding to the natural language processing model based on the historical interaction data.

[0253] The optimization module is used to input optimized training samples into the natural language processing model to retrain the student network until the adaptive optimization total loss function meets the preset convergence requirements, so as to obtain the optimized natural language processing model.

[0254] The update module is used to collect new text information input by the target user and input the new text information into the natural language processing optimization model to output corresponding new response information.

[0255] Optionally, in one embodiment of this application, the text message response device 10 further includes: a historical data acquisition module, a key index calculation module, and a key index comparison module.

[0256] The historical data acquisition module is used to collect historical text input data and historical response information corresponding to the natural language processing model before selecting a preset number of teacher network layers that meet the preset network layer contribution requirements from the teacher network of the preset natural language processing model.

[0257] The criticality index calculation module is used to determine the contribution of different network layers in the natural language processing model based on historical text input data and historical response information, and to calculate the relative criticality index of different network layers in the teacher network and student network according to the contribution of the network layers.

[0258] The criticality comparison module is used to compare relative criticality indicators with preset criticality thresholds to determine the number of indicators with relative criticality greater than the criticality threshold, and to determine the preset number based on the number of indicators.

[0259] For a description of the features in the embodiment corresponding to the text message response device, please refer to the relevant description of the embodiment corresponding to the text message response method, which will not be repeated here.

[0260] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described text message response method embodiments.

[0261] Embodiments of this application also provide a non-volatile computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described text information response method embodiments when running.

[0262] In one exemplary embodiment, the aforementioned non-volatile computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0263] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described text information response method embodiments.

[0264] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described text information response method embodiments.

[0265] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0266] The foregoing has provided a detailed description of a text information response method, apparatus, device, and medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A text message response method, characterized in that, Includes the following steps: A preset number of teacher network layers that meet the preset network layer contribution requirements are selected from the teacher network of the preset natural language processing model, and the preset number of student network layers that meet the preset network layer contribution requirements are selected from the student network of the natural language processing model. Network layer matching is performed on the teacher network layers and the student network layers to construct a corresponding distillation layer set. Obtain the attention graph sets of the teacher network layer and student network layer in any distillation layer of the distillation layer set, and group and integrate the attention graph sets to obtain attention graph groups that meet the preset quantity requirements. Then, perform column-wise concatenation operation on the attention graphs in the attention graph groups to generate the corresponding teacher aggregation matrix and student aggregation matrix. The matrix dimensions corresponding to the teacher aggregation matrix and the student aggregation matrix are determined respectively, so as to construct the learnable transformation matrices corresponding to the teacher aggregation matrix and the student aggregation matrix according to the matrix dimensions. Matrix multiplication operations are then performed on the teacher aggregation matrix and the student aggregation matrix with the corresponding learnable transformation matrices to obtain the corresponding teacher feature matrix and student feature matrix. Based on the teacher feature matrix and the student feature matrix, a double distillation loss function corresponding to the natural language processing model is determined. The student network is then trained based on the double distillation loss function. The trained student network is then used to infer the text information input by the target user and generate the corresponding response information.

2. The text message response method according to claim 1, characterized in that, The step of obtaining the attention map sets of the teacher network layer and the student network layer in any distillation layer of the distillation layer set, and grouping and integrating the attention map sets to obtain attention map groups that meet a preset quantity requirement, includes: Based on the multi-head attention mechanism in the teacher network layer, multiple corresponding teacher layer attention graphs are generated, and using the multi-head attention mechanism in the student network layer, multiple corresponding student layer attention graphs are generated. The generation sequence numbers corresponding to the multiple teacher-level attention maps and the multiple student-level attention maps are determined respectively, and the multiple teacher-level attention maps and the multiple student-level attention maps are sequentially divided into corresponding attention map groups according to the generation sequence numbers, so as to obtain attention map groups that meet the preset quantity requirements.

3. The text message response method according to claim 2, characterized in that, The step of determining the double distillation loss function corresponding to the natural language processing model based on the teacher feature matrix and the student feature matrix includes: Normalization calculations are performed on the teacher feature matrix and the student feature matrix respectively to obtain the corresponding attention probability distribution matrix, and downsampling operations are performed on the teacher feature matrix and the student feature matrix to obtain the corresponding downsampling matrix; Based on the attention probability distribution matrix and the downsampling matrix, calculate the fine-grained distribution loss and coarse-grained structure loss corresponding to the natural language processing model. The fine-grained distribution loss and the coarse-grained structure loss of the coverage are weighted and summed to obtain the double distillation loss function of the natural language processing model.

4. The text message response method according to claim 1, characterized in that, The step of determining the double distillation loss function corresponding to the natural language processing model based on the teacher feature matrix and the student feature matrix, and training the student network based on the double distillation loss function, includes: Based on the double distillation loss function, calculate the total single-layer loss of the distillation layer corresponding to the attention map group; Based on the single-layer total loss, calculate the mean of the single-layer total loss corresponding to the set of distillation layers, and determine the distillation loss function corresponding to the natural language processing model based on the mean of the single-layer total loss. Obtain the input word sequence of the natural language processing model, and determine the probability distribution vector of words at different positions in the input word sequence belonging to different words in a preset word table, so as to determine the cross-entropy loss function corresponding to the natural language processing model based on the probability distribution vector; Based on the distillation loss function and the cross-entropy loss function, a total loss function corresponding to the natural language processing model is constructed, and the student network is trained using the total loss function.

5. The text message response method according to claim 4, characterized in that, The process of training the student network based on the double distillation loss function, and then using the trained student network to infer the text information input by the target user and generate corresponding response information, includes: A preset text database is sampled to obtain multiple text sample data, and the multiple text sample data are preprocessed to obtain multiple corresponding text sample standard data. The training dataset of the natural language processing model is constructed using the multiple text sample standard data. The teacher network of the natural language processing model is trained using the training dataset to obtain the trained teacher network. Based on the trained teacher network, the network architecture of the student network is determined, and the weight matrix and model parameters of the student network are initialized to obtain the initial weight matrix and initial model parameters corresponding to the student network. Based on the training dataset, and in combination with the initial weight matrix, the initial parameters of the model, and the network architecture, the student network is trained in multiple rounds of iterative training. After each round of training, the order of multiple text sampling standard data in the training dataset is changed according to a preset transformation strategy to construct a corresponding disordered training dataset. In the next round of training, the student network is trained in multiple batches using the disordered training dataset. In each batch of training, the total loss function is calculated to obtain the corresponding total loss value. Training stops when the total loss value meets a preset convergence requirement to obtain the trained student network. The system obtains text information input by the target user, inputs the text information into a trained student network, and uses the trained student network to infer the text information in order to generate response information corresponding to the text information.

6. The text message response method according to claim 3, characterized in that, The calculation of the fine-grained distribution loss and coarse-grained structure loss corresponding to the natural language processing model based on the attention probability distribution matrix and the downsampling matrix includes: The teacher feature matrix and the student feature matrix are normalized respectively to obtain the corresponding attention probability distribution matrix; Compare the elements in the attention probability distribution matrix with a preset probability value threshold to determine multiple target elements that are greater than the probability value threshold, and perform an element-by-element XOR operation on the multiple target elements to obtain a first operation result; The elements in the attention probability distribution matrix other than the multiple target elements are subjected to probability decay to obtain the corresponding decay probability values, and the decay probability values ​​are subjected to element-by-element XOR operation to obtain the corresponding second operation result. Based on the first calculation result and the second calculation result, the XOR operation result corresponding to the attention probability distribution matrix is ​​determined, so as to calculate the fine-grained distribution loss of the coverage according to the XOR operation result.

7. The text message response method according to claim 3, characterized in that, The step of calculating the fine-grained distribution loss and coarse-grained structure loss corresponding to the natural language processing model based on the attention probability distribution matrix and the downsampling matrix further includes: Perform downsampling operations on the teacher feature matrix and the student feature matrix to obtain the corresponding downsampling matrix; The average value of the downsampling matrices corresponding to the teacher feature matrix and the student feature matrix is ​​calculated respectively. Elements in the downsampling matrix that are greater than the average value are set as first target values, and elements in the downsampling matrix that are less than or equal to the average value are set as second target values. Based on the first target value and the second target value, the binarized sampling matrix of the downsampling matrix corresponding to the teacher feature matrix and the student feature matrix is ​​determined. An element-wise XOR operation is performed on the binary sampling matrices corresponding to the teacher feature matrix and the student feature matrix to obtain the difference quantization matrix between the teacher feature matrix and the student feature matrix. The difference quantization matrix is ​​then subjected to average pooling to obtain the coarse-grained structure loss.

8. The text message response method according to claim 4, characterized in that, The step of obtaining the input lexical sequence of the natural language processing model and determining the probability distribution vectors of lexical units at different positions in the input lexical sequence belonging to different lexical units in a preset lexical table, and determining the cross-entropy loss function corresponding to the natural language processing model based on the probability distribution vectors, includes: The natural language processing model is used to obtain the probability distribution vector of the next word in the current position of the input word sequence belonging to different words in the word table, and the logarithm of the probability distribution vector is calculated. Determine the actual word category corresponding to the current position in the input word sequence, and obtain the cross-entropy loss function based on the actual word category and the logarithm.

9. The text message response method according to claim 4, characterized in that, The construction of the total loss function corresponding to the natural language processing model based on the distillation loss function and the cross-entropy loss function includes: Determine the model processing requirements corresponding to the natural language processing model, and calculate the weight coefficients corresponding to the distillation loss function based on the model processing requirements; Based on the weighting coefficients, a linear operation is performed on the distillation loss function and the cross-entropy loss function to obtain the total loss function.

10. The text message response method according to claim 1, characterized in that, The step of obtaining the attention map sets of the teacher network layer and the student network layer in any distillation layer of the distillation layer set, and grouping and integrating the attention map sets to obtain attention map groups that meet a preset quantity requirement, further includes: Determine whether the number of attention graphs in the attention graph group corresponding to the teacher network layer and the attention graph group corresponding to the student network layer are the same; If the number of attention graphs in the attention graph group corresponding to the teacher network layer and the attention graph group corresponding to the student network layer are the same, then the same number of attention graph groups are constructed for the teacher network layer and the student network layer respectively. If the number of attention graphs in the attention graph group corresponding to the teacher network layer and the attention graph group corresponding to the student network layer are not the same, then the number of attention graphs in the last attention graph group corresponding to the teacher network layer and the student network layer is adjusted to obtain an attention graph group that satisfies the preset principle of equal number of attention graphs.

11. The text message response method according to claim 1, characterized in that, After using the trained student network to infer the text information input by the target user and generate the corresponding response information, the process further includes: The reply information is sent to the user terminal, and after the target user receives the reply information, the multimodal interaction feedback data corresponding to the target user is obtained; The multimodal interactive feedback data is subjected to multimodal fusion processing to construct corresponding dynamic feedback weight coefficients. Based on the dynamic feedback weight coefficients, the weight coefficients of the distillation loss function are adjusted. The distillation loss function and the cross-entropy loss function are then weighted and summed again according to the adjusted weight coefficients of the distillation loss function to construct the adaptive optimization total loss function corresponding to the natural language processing model. Obtain historical interaction data corresponding to the natural language processing model, and construct optimized training samples corresponding to the natural language processing model based on the historical interaction data; The optimized training samples are input into the natural language processing model to retrain the student network until the adaptive optimization total loss function meets the preset convergence requirement, so as to obtain the natural language processing optimized model. The new text information input by the target user is collected and input into the natural language processing optimization model to output corresponding new response information.

12. The text message response method according to claim 1, characterized in that, Before selecting a predetermined number of teacher network layers that meet the predetermined contribution requirements from the teacher network of the predetermined natural language processing model, the following steps are also included: Collect historical text input data corresponding to the natural language processing model and historical response information corresponding to the historical text input data; Based on the historical text input data and the historical response information, the network layer contribution of different network layers corresponding to the natural language processing model is determined, and the relative criticality index of different network layers in the teacher network and the student network is calculated according to the network layer contribution. The relative criticality index is compared with a preset criticality threshold to determine the number of relative criticality indices that are greater than the criticality threshold, and the preset number is determined based on the number of indices.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the text message response method as described in any one of claims 1 to 12 when executing the computer program.

Images