Natural language processing method, electronic device, storage medium and program product

Abstract

The present application discloses a natural language processing method, electronic device, storage medium and program product, and relates to the field of computer technology. The present application generates a student network model including a plurality of zero vector column weight matrices through a knowledge distillation mechanism, so that when storing the student network model, the zero vector column in the weight matrix can be omitted, thereby reducing storage requirements and saving a large amount of storage space. When performing inference operations on text data through the stored student network model, the zero vector column does not participate in the operation, thereby reducing the amount of calculation, improving the inference operation speed, and thus improving the overall operation efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a natural language processing method, electronic device, storage medium, and program product. Background Art

[0002] Against the backdrop of the current booming development of artificial intelligence technology, the Transformer architecture model (a deep learning model architecture used for natural language processing and other sequence-to-sequence tasks) has demonstrated excellent performance and broad application prospects in many fields. However, with the continuous expansion of model scale and the increasing complexity of application scenarios, its storage requirements have also become a key issue that needs to be urgently addressed. In the storage link, the relevant model storage method requires the complete storage of the student model weight matrix after knowledge distillation, which often requires a large amount of storage space. This not only places high demands on storage devices, but also, to a certain extent, limits the application and deployment of the model in resource-constrained environments. In addition, during the model inference calculation process, the entire weight matrix of the stored student model is involved in the calculation, which results in a huge amount of calculation and low computational efficiency. Summary of the Invention

[0003] The present application provides a natural language processing method, electronic device, storage medium and program product to at least solve the problems of large storage resource usage and low reasoning operation efficiency in related technologies.

[0004] This application provides a natural language processing method, including:

[0005] Get a pre-built teacher network model;

[0006] Based on the teacher network model and the knowledge distillation mechanism, a target student network model is generated. The target student network model includes at least a weight matrix, and the weight matrix includes at least one zero vector column.

[0007] In response to receiving a natural language processing request, obtaining text data and dividing the text data into a plurality of tokens;

[0008] The word unit is input into the target student network model, and the word unit is inferred and calculated based on the weight matrix including the zero vector column in the target student network model to determine the output result corresponding to the text data.

[0009] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the following steps of the natural language processing method when executing the computer program:

[0010] Get a pre-built teacher network model;

[0011] Based on the teacher network model and the knowledge distillation mechanism, a target student network model is generated. The target student network model includes at least a weight matrix, and the weight matrix includes at least one zero vector column.

[0012] In response to receiving a natural language processing request, obtaining text data and dividing the text data into a plurality of tokens;

[0013] The word unit is input into the target student network model, and the word unit is inferred and calculated based on the weight matrix including the zero vector column in the target student network model to determine the output result corresponding to the text data.

[0014] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the following steps of the natural language processing method are implemented:

[0015] Get a pre-built teacher network model;

[0016] Based on the teacher network model and the knowledge distillation mechanism, a target student network model is generated. The target student network model includes at least a weight matrix, and the weight matrix includes at least one zero vector column.

[0017] In response to receiving a natural language processing request, obtaining text data and dividing the text data into a plurality of tokens;

[0018] The word unit is input into the target student network model, and the word unit is inferred and calculated based on the weight matrix including the zero vector column in the target student network model to determine the output result corresponding to the text data.

[0019] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the following steps of the natural language processing method:

[0020] Get a pre-built teacher network model;

[0021] Based on the teacher network model and the knowledge distillation mechanism, a target student network model is generated. The target student network model includes at least a weight matrix, and the weight matrix includes at least one zero vector column.

[0022] In response to receiving a natural language processing request, obtaining text data and dividing the text data into a plurality of tokens;

[0023] The word unit is input into the target student network model, and the word unit is inferred and calculated based on the weight matrix including the zero vector column in the target student network model to determine the output result corresponding to the text data.

[0024] This application uses a knowledge distillation mechanism to generate a student network model including a weight matrix of multiple zero vector columns, so that when storing the student network model, the zero vector column in the weight matrix can be omitted, reducing storage requirements and saving a large amount of storage space. When performing inference operations on text data through the stored student network model, the zero vector column does not participate in the operation, reducing the amount of calculation, improving the inference operation speed, and thus improving the overall operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 An application environment diagram of a natural language processing method is provided for an embodiment of the present application;

[0027] Figure 2 A schematic diagram of the overall process of a natural language processing method is provided for an embodiment of the present application;

[0028] Figure 3 A schematic diagram of the workflow of a query embedding generation module of a natural language processing method is provided for an embodiment of the present application;

[0029] Figure 4 A schematic diagram of the workflow of a key embedding generation module of a natural language processing method is provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of the workflow of a value embedding generation module of a natural language processing method is provided for an embodiment of the present application;

[0031] Figure 6 A schematic diagram of the overall structure of a single-head attention module of a natural language processing method is provided in an embodiment of the present application;

[0032] Figure 7 A schematic diagram of the workflow of a feedforward neural network layer of a natural language processing method is provided for an embodiment of the present application;

[0033] Figure 8 A schematic diagram of a distillation loss process between a teacher network model and a student network model of a natural language processing method is provided for an embodiment of the present application;

[0034] Figure 9 FIG. 1 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0035] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0036] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0037] It should be noted that the terms "S1", "S2", etc. are used only for the purpose of describing the steps and do not specifically refer to the order or sequence, nor are they used to limit this application. They are merely for the convenience of describing the method of this application and should not be understood as indicating the order of the steps. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0038] According to the background technology, as the scale of models continues to expand and the application scenarios become increasingly complex, their storage requirements have also become a key issue that needs to be solved urgently. On the one hand, the storage method of related models often requires a large amount of storage space, which not only puts high demands on storage devices, but also limits the application and deployment of models in resource-constrained environments to a certain extent. In order to meet this challenge, low-bit quantization methods have emerged. By quantizing the values ​​in the model with low bits, such as from 32-bit floating point quantization to 8 bits or even lower bits, the space required to store each parameter can be significantly reduced. This method effectively compresses the storage size of the model without seriously affecting the performance of the model, making it possible to store more data in the same storage space. On the other hand, low-order matrix expression has also become an important means to reduce storage requirements. In many cases, the weight matrix in the model has certain redundancy and structure. By using low-order matrix expression technology, the original high-dimensional and complex weight matrix can be decomposed into a combination of low-order matrices. In this way, only the relevant information of these low-order matrices needs to be saved during storage, which reduces the storage overhead compared to directly storing the original weight matrix. However, the above technology still needs to save the complete student network model weight matrix, which takes up a lot of storage space. In the subsequent inference operation process, the entire weight matrix of the student model is involved in the operation, and the amount of calculation is huge.

[0039] To solve the above technical problems, the present application provides a natural language processing method, an electronic device, a storage medium and a program product. Through the knowledge distillation mechanism, a student network model including a weight matrix of multiple zero vector columns is generated, so that when storing the student network model, the zero vector column in the weight matrix can be omitted, reducing the storage requirements and saving a lot of storage space. When the text data is inferred through the stored student network model, the zero vector column does not participate in the operation, which reduces the amount of calculation, improves the speed of inference operation, and thus improves the overall operation efficiency.

[0040] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0041] The natural language processing method provided in this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with a data processing platform provided on the server 104 via a network. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0042] To help those skilled in the art better understand the present application, the Transformer architecture model involved in this application is explained, including:

[0043] (1) Query embedding generation module;

[0044] like Figure 3 As shown, the query embedding generation module embeds its input into ( ) is converted to a query embedding ( ), assuming the input embedding yes A column matrix with 1 row and 1 column, that is ; Query Embedding yes A column matrix with 1 row and 1 column, that is ; The query embedding generation module contains the weight matrix yes OK, The matrix of columns, that is , then the process of input embedding passing through the query embedding generation module to output query embedding can be formulated as: .

[0045] (2) Key embedding generation module;

[0046] like Figure 4 As shown, the key embedding generation module embeds its input into ( ) is converted to a key embedding ( ), assuming the input embedding yes A column matrix with 1 row and 1 column, that is ;Key embedding yes A column matrix with 1 row and 1 column, that is ;The key embedding generation module contains the weight matrix yes OK, The matrix of columns, that is , then the process of input embedding passing through the key embedding generation module to output key embedding can be formulated as: .

[0047] (3) Value embedding generation module;

[0048] like Figure 5 As shown, the value embedding generation module embeds its input into ( ) is converted to a value embedding ( ), assuming the input embedding yes A column matrix with 1 row and 1 column, that is ; Value embedding yes A column matrix with 1 row and 1 column, that is ; The weight matrix included in the value embedding generation module yes OK, The matrix of columns, that is , then the process of input embedding passing through the value embedding generation module to output value embedding can be formulated as: .

[0049] Among them, such as Figure 6 As shown in Figure 2, within the Transformer-based model, the query embedding generation module, key embedding generation module, and value embedding generation module are always used as a whole, sharing the same input embedding ( ), when using single-head attention, the input embedding ( ) are input into the query embedding generation module, key embedding generation module and value embedding generation module respectively; In addition, when multi-head attention is adopted, multiple groups of query embedding generation modules, key embedding generation modules and value embedding generation modules are included, and the input embedding ( ) are input into each set of query embedding generation module, key embedding generation module and value embedding generation module respectively.

[0050] It should be noted that the input embedding (X) of the above module can be the word embedding generated by the word embedding encoder when the word is input into the model, or it can be the embedding within the model. Taking text input as an example, when text data is input into the model, the text data is divided into several words, and then each word is input into the word embedding encoder to generate a word embedding. When representing the above word embedding, then usually ;when When representing the internal embedding of the model, , it should be noted that, and The numerical relationship does not affect the application scope of this application.

[0051] (4) Feedforward neural network layer (also called feedforward neural network module);

[0052] like Figure 7 As shown, the feedforward neural network layer embeds its input into ( ) is converted to an output embedding ( ). Assume that the input embedding yes A column matrix with 1 row and 1 column, that is ; Output embedding yes A column matrix with 1 row and 1 column, that is ; The weight matrix contained in the feedforward neural network layer yes OK, The matrix of columns, that is .So, In this embodiment, the feedforward neural network layer refers to the feedforward neural network layer inside each Transformer module, and its input embedding and output embedding The dimensions are the same, i.e. .

[0053] like Figure 2 As shown, the embodiment of the present application provides a natural language processing method, which is applied to Figure 1 The following steps are used as an example to illustrate the terminal in the figure:

[0054] S1: Get the pre-built teacher network model.

[0055] It should be noted that the teacher network model refers to a large, complex deep neural network model with a large number of pre-trained parameters in the knowledge distillation process. The teacher network model is usually trained on a large dataset, has high complexity and parameter count, and can perform well on specific tasks. In this application, the teacher network model refers to a model based on the Transformer architecture. , also called source model. ‌

[0056] S2: Based on the teacher network model and the knowledge distillation mechanism, a target student network model is generated. The target student network model includes at least a weight matrix, and the weight matrix includes at least one zero vector column.

[0057] It should be noted that the knowledge distillation mechanism refers to the process of transferring "knowledge" from the teacher model with a very large number of parameters to the lightweight student network model. After the knowledge distillation process, the student network model can show performance comparable to that of the teacher network model. In this application, the target student network model refers to the model based on the Transformer architecture. ,Model and model Have the same network structure, that is, the dimensions of the weight matrices at the corresponding positions of the two are the same; this application uses a neural network training method to train the source model Distillation to obtain the target model , the neural network training loss function is , It consists of two parts, namely the distillation loss function and task-related loss function , The distillation loss function is and task-related loss function is a function of the independent variable, , the weight matrix of the target student network model obtained after processing by the knowledge distillation mechanism contains multiple zero vector columns; among them, the teacher network model and the student network model are both large language models, such as the Question-Answer large language model, the question is input into the model, and the output is the answer corresponding to the question.

[0058] S3: In response to receiving the natural language processing request, obtain text data and divide the text data into multiple word units.

[0059] It should be noted that natural language processing requests are generally issued by the task node that stores the target student model. The request includes text data. For example, tokens are used to divide text data into smaller units. These units may be words, characters, or subwords. Tokens are the most basic units used by language models such as GPT (Generative Pre-trained Transformer) during training and inference.

[0060] S4: Input the word unit into the target student network model, and perform inference calculation on the word unit based on the weight matrix including the zero vector column in the target student network model to determine the output result corresponding to the text data.

[0061] It should be noted that when performing inference calculations on word units, the zero vector column does not participate in the inference calculation process. The text data is the input question-related data, and the output result is the answer to the question.

[0062] In the above embodiment, a student network model including a weight matrix of multiple zero vector columns is generated through the knowledge distillation mechanism, so that when storing the student network model, the zero vector column in the weight matrix can be omitted, reducing the storage requirements and saving a lot of storage space. When the text data is inferred through the stored student network model, the zero vector column does not participate in the operation, which reduces the amount of calculation, improves the speed of inference operation, and thus improves the overall operation efficiency.

[0063] In some specific implementations, generating a target student network model based on the teacher network model and the knowledge distillation mechanism includes:

[0064] Based on the teacher network model, the first student network model is constructed, wherein represents the weight matrix in the first student network model, , represents the first diagonal matrix, Represents a randomly initialized matrix, where the teacher network model and the first student network model have the same network structure. The first diagonal matrix refers to a matrix in which all element values ​​​​at other positions are zero except for the element values ​​​​on the main diagonal. The element values ​​of the randomly initialized matrix will be updated during the training process;

[0065] Based on the teacher network model and the first student network model, a training loss function is constructed. The training loss function is a function with the distillation loss function and the task-related loss function as independent variables;

[0066] Obtain training data, where training data refers to text data. The training data of this application collects a large-scale text data set, containing hundreds of millions of sentences, including sentences in multiple languages. For example, Chinese sentences account for 30%, English sentences account for 60%, and other languages ​​account for 10%. During the model training process, text data is sampled from the large-scale text data set by random sampling. The language type ratio is an exemplary value. Using other numerical ratios does not affect the effect of this application. The text data is input into the large prediction model to generate intermediate state values ​​(such as query embedding, key embedding, and value embedding, etc.) during the model operation, as well as the final output text. In order to prevent the order of training samples in the training data set D from affecting the performance of the model, at the beginning of the model training, all samples in the training data set D are randomly arranged. In each round of training, the model training algorithm reads a batch of samples, such as 1024 samples;

[0067] Based on the training loss function and the training data, the first student network model is trained to update the weight matrix in the first student network model;

[0068] In response to the current training cycle being the preset training cycle, the training is ended to obtain a second student network model, wherein the second student network model is obtained by using represents the weight matrix in the second student network model, , represents the randomly initialized matrix after training update;

[0069] Further, the diagonal elements in the first diagonal matrix are sorted in order of size, such as sorting the diagonal elements in the first diagonal matrix in order from largest to smallest;

[0070] According to the preset ratio, multiple element values ​​are selected from the sorting results, and the other element values ​​that are not selected are set to zero to generate the second diagonal matrix , where the preset ratio can be set according to actual needs, and its value range is 30% -70%. The preferred value can be set to 50%, that is, retaining the first 50% of the diagonal elements in the sorting result, and setting the other element values ​​in the first diagonal matrix to zero to generate the second diagonal matrix ;

[0071] Based on the second diagonal matrix and The product of , determines the weight matrix in the third student network model as ;

[0072] Fix the zero vector column in the weight matrix of the third student network model and set the element value state in the remaining columns to the learnable state;

[0073] Based on the task-related loss function and the text data corresponding to the target task node, the third student network model is trained to update the element values ​​in the remaining columns, where the target task node is the node corresponding to a more specific task scenario;

[0074] In response to the training being completed, a target student network model is generated.

[0075] In the above embodiment, by constructing a teacher network model and a student network model with the same network structure, and training the student network model by setting a diagonal matrix and a training loss function, the obtained student network model presents a unique characteristic in the weight matrix dimension, that is, there are a large number of columns with zero element values ​​in the weight matrix. Based on this, the space required for storing the student network model can be greatly reduced, and the hardware storage cost is reduced. On the other hand, during the model inference operation, the existence of zero-value columns can effectively reduce the amount of calculation, significantly improve the inference speed, and thus improve the overall computing efficiency.

[0076] In some specific implementations, after generating the target student network model, the method further includes:

[0077] The target student network model is deployed to the target task node, that is, the target student network model is deployed to the corresponding task node through which the model is trained by the text data of the task node. When deploying and storing the target student network model, there is no need to save the zero vector columns, only the location information of these columns needs to be recorded.

[0078] In the above implementation, by deploying a more targeted student network model to the corresponding task node, the request of the target task node can be processed efficiently and accurately, thereby improving the task processing efficiency.

[0079] In some specific embodiments, the above method further comprises:

[0080] Get the text data corresponding to the historical tasks of multiple task nodes;

[0081] Calculate the correlation between the two task nodes based on the text data, including:

[0082] Acquire first text data corresponding to the historical task of the first task node and second text data corresponding to the historical task of the second task node;

[0083] Calculate the similarity between the first text data and the second text data based on a similarity algorithm, wherein the similarity algorithm can be an algorithm based on string matching, an algorithm based on a bag-of-words model, or the like for calculating text similarity, which is a commonly used algorithm and the specific calculation process is not repeated here;

[0084] Based on the similarity, the correlation between the first task node and the second task node is evaluated, that is, the higher the similarity, the higher the correlation;

[0085] In response to the similarity being greater than a preset threshold, it is determined that the first task node and the second task node are highly correlated. Based on this, the same student network model is deployed at the first task node and the second task node to process corresponding tasks. The preset threshold can be set according to actual needs, such as 90%.

[0086] In the above implementation, by calculating the correlation between task nodes and deploying the same student network model on two task nodes with high correlation, the number of model training times can be reduced and the overall work efficiency of the system can be improved.

[0087] In some specific embodiments, training the first student network model based on the training loss function and the training data to update the weight matrix in the first student network model includes:

[0088] The weight matrix parameters of the teacher network model are fixed, and the element values ​​and other parameters in the first student network model are randomly initialized. The other parameters refer to the parameters used to train the network model. The element values ​​in the first student network model are randomly initialized based on the Gaussian distribution. The mean of the element values ​​of the weight matrix after the processing is 0 and the standard deviation is 0.01.

[0089] Set the relevant parameters of the stochastic gradient descent algorithm and define the current training cycle as The total training cycle is , in response to the , perform the following operations, where the stochastic gradient descent algorithm is the Adam algorithm, the parameters related to the stochastic gradient descent algorithm and the total training cycle can be set according to actual needs. For example, the total training cycle is 30, and the parameters related to the stochastic gradient descent algorithm are the learning rate is , momentum coefficient , , , the learning rate strategy is the cosine strategy, and the number of samples in a mini-batch is B = 1024;

[0090] For the current training cycle Perform an addition operation, that is, ;

[0091] Randomly shuffle the order of samples in the training data and select the target batch of training samples from the training data. The order of samples in the data set is randomly shuffled. Select a batch B of training samples;

[0092] According to the training samples and the parameters related to the stochastic gradient descent algorithm, the training operation is performed, the training loss value is calculated, and the weight matrix in the first student network model is updated using the stochastic gradient descent algorithm;

[0093] Repeating the sample selection and weight matrix update steps, and ending the current training cycle in response to all training samples in the training data being utilized;

[0094] In response to the current training cycle being the preset training cycle, the training is ended to obtain a second student network model.

[0095] In the above implementation, the model is trained using the above model training method to improve the performance of the model.

[0096] In some specific embodiments, constructing a training loss function includes:

[0097] Based on the distillation loss function and the task-related loss function, a training loss function is constructed.

[0098] Among them, the training loss function includes:

[0099] ;

[0100] in, represents the training loss value, represents the task-related loss value, Represents a constant parameter, Represents the distillation loss value.

[0101] The distillation loss function includes:

[0102] ;

[0103] ;

[0104] in, represents the distillation loss value, represents the number of weight matrices in the teacher network model, Representation matrix and matrix The distillation loss value between represents a random matrix, represents a diagonal matrix.

[0105] matrix and matrix The calculation method of the distillation loss value between includes:

[0106] ;

[0107] in, express norm, represents the norm of matrix F, Represents a constant parameter, A vector representing the diagonal elements of a diagonal matrix.

[0108] Task-related loss functions include:

[0109] ;

[0110] in, represents the training loss value, Indicates the number of words, Represents a vector No. elements, Represents a vector No. elements, express dimensional one-hot vector, representing the position The category corresponding to the real word at Indicates the predicted dimensional probability distribution vector.

[0111] Specifically, the distillation loss function The definition includes: Figure 8 As shown, without loss of generality, assuming that the source model Any weight matrix in ,and Corresponding target model The weight matrix in is , both have the same input, where the weight matrix and the weight matrix Represents the weight matrix of any embedding generation module, the weight matrix of the key embedding generation module, the weight matrix of the value embedding generation module, or the weight matrix of the feedforward neural network layer. This application requires that for the source model Any weight matrix in and target model The corresponding weight matrix in , when their input data are the same, the outputs generated are also the same or similar. Any weight matrix in With target model The corresponding weight matrix in , under the premise of consistent input data, the output results of the two must strictly meet the consistency or approximation requirements, that is, the outputs produced by the two are either exactly the same or infinitely close, so as to ensure the accuracy and effectiveness of the model knowledge transfer process, so that the weight matrix and the weight matrix Methods with outputs that are infinitely close or identical are as follows:

[0112] Assume the weight matrix and the weight matrix The input data is the same as that of , column vector Input to the weight matrix , and get the output column vector , column vector Input to the weight matrix , and get the output column vector , at the same input In this case, the output is the same or close to the result, that is, the requirement and Same or close, in this application, use measure and The similarity between When it is zero, and same, The smaller, and The closer (similar), the optimization goal of the model training algorithm is to minimize , express and The square of the L2 norm of the difference, where express norm;

[0113] ;

[0114] Assumptions The element values ​​follow the Gaussian distribution, minimizing and The square of the L2 norm of the difference can be transformed into minimizing and The square of the F norm of the difference is as follows:

[0115] ;

[0116] Based on this, the optimization objective is transformed into ,in represents minimization, is the weight matrix of the source model, and the optimization goal is to change , thereby minimizing ,From the perspective of the entire optimization process, the weight matrix of the source model remains unchanged, and the weight matrix of the target model is optimized; At the same time, this application also requires If all the elements of some columns are zero, then these columns do not need to be saved during the model saving process, and only the position information of these columns needs to be recorded. In order to achieve this optimization goal, this application further It is represented as the multiplication of two matrices, ,in It is a diagonal matrix (except for the elements on the main diagonal, the elements in other positions are all zero);

[0117] ;

[0118] To achieve The optimization goal of all the elements of some columns are zero, which requires a diagonal matrix Some elements on the main diagonal of are zero, if the diagonal matrix If some elements on the main diagonal of are zero, then and Calculated All elements of some columns in are zero. In order to achieve the above goal, extract the diagonal matrix The diagonal elements of , then, the vector of The norm is added as a constraint to the following formula. By adding the above constraint, it is possible to make the diagonal matrix The effect of returning some diagonal elements to zero, based on the above analysis and demonstration, the training weight matrix The loss function is defined as :

[0119] ;

[0120] in, is a constant coefficient, preferably, ;

[0121] In the above expression, using norm as a constraint to find the optimal sparse term, but The problem of minimizing the norm is NP-hard, so The optimization problem is relaxed to a higher-dimensional norm problem, such as norm;

[0122] ;

[0123] The above analysis of a set of matrices and The distillation loss between them can be expanded by the above analysis to obtain the distillation loss function between the models. The expression is as follows:

[0124] ;

[0125] in, , Representation source model The number of weight matrices in the target model The number of weight matrices in .

[0126] Furthermore, the task-related loss function is defined as follows: knowledge distillation is performed on a large language model. The task is to predict the next token given a sequence of tokens. Then, the task loss function is Using the cross entropy loss function, assuming there is a A sequence of word units, for each position , the model needs to predict the next word belongs to the vocabulary The probability distribution of words, let is a A one-hot vector of dimension denoting the position The category corresponding to the real word at The model predicts dimensional probability distribution vector, then the cross entropy loss function can be expressed as:

[0127] ;

[0128] in, represents the training loss value, Indicates the number of words, Represents a vector No. elements, Represents a vector No. elements, express dimensional one-hot vector, representing the position The category corresponding to the real word at Indicates the predicted dimensional probability distribution vector.

[0129] Based on the above distillation loss function and task-related loss function, the overall loss function for model training can be constructed as follows:

[0130] ;

[0131] in, The distillation loss function is And task-related loss function is a function of the independent variable. In this application, The linear function is as follows:

[0132] ;

[0133] in, is a constant coefficient used to balance and Contribution, preferably, , by adjusting Better accuracy can be achieved.

[0134] In the above embodiment, the overall training loss function is constructed by constructing the task-related loss function and the distillation loss function to calculate the loss value, and then the parameters of the large language model to be trained are updated, which can improve the performance of the large language model.

[0135] In some specific implementations, in response to receiving a natural language processing request, obtaining text data, and dividing the text data into a plurality of tokens includes:

[0136] Receive natural language processing requests from target task nodes;

[0137] In response to receiving the natural language processing request, parsing the natural language processing request to obtain text data corresponding to the natural language processing request;

[0138] According to the division rules, the text data is divided to obtain multiple word units corresponding to the text data. The division rules can be set according to actual needs, such as formulating a series of rules for division according to the grammatical rules and vocabulary characteristics of the language. For example, in Chinese, individual Chinese characters and words can be identified according to the dictionary, and some word formation rules can also be combined, such as "prefix + root", "root + suffix" and other patterns.

[0139] In the above implementation, by dividing text data into word units, the model learning efficiency can be improved and the model generalization ability can be enhanced.

[0140] In the above-mentioned natural language processing method, the method includes: obtaining a pre-built teacher network model; generating a target student network model based on the teacher network model and a knowledge distillation mechanism, the target student network model including at least one weight matrix, and the weight matrix including at least one zero vector column; in response to receiving a natural language processing request, obtaining text data and dividing the text data into multiple word units; inputting the word units into the target student network model, and performing inference calculations on the word units based on the weight matrix including the zero vector column in the target student network model to determine the output results corresponding to the text data. This application uses a knowledge distillation mechanism to generate a student network model including multiple zero vector column weight matrices. In terms of storage, by omitting the zero vector column of the student network model weight matrix, the storage cost is greatly reduced, and the storage capacity of complex models can be reduced by up to 50%, which is more suitable for devices with limited storage resources. In terms of calculation, the zero vector column does not participate in the calculation during inference, and the calculation amount is reduced to 30%-50% of the traditional method. The inference speed is significantly improved, greatly enhancing the real-time performance of the system. In scenarios with high response speed requirements such as autonomous driving and intelligent security, it can quickly process data and make accurate decisions, and has broad application prospects.

[0141] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0142] It should be understood that although Figure 2-Figure 8 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 2-Figure 8At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0143] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a natural language processing method is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0144] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0145] An embodiment of the present application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform steps in an embodiment of a natural language processing method, including:

[0146] S1: Get the pre-built teacher network model;

[0147] S2: Based on the teacher network model and the knowledge distillation mechanism, a target student network model is generated. The target student network model includes at least a weight matrix, and the weight matrix includes at least one zero vector column.

[0148] S3: In response to receiving the natural language processing request, obtaining text data and dividing the text data into a plurality of tokens;

[0149] S4: Input the word unit into the target student network model, and perform inference calculation on the word unit based on the weight matrix including the zero vector column in the target student network model to determine the output result corresponding to the text data.

[0150] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the steps of the embodiment of the natural language processing method when run, including:

[0151] S1: Get the pre-built teacher network model;

[0152] S2: Based on the teacher network model and the knowledge distillation mechanism, a target student network model is generated. The target student network model includes at least a weight matrix, and the weight matrix includes at least one zero vector column.

[0153] S3: In response to receiving the natural language processing request, obtaining text data and dividing the text data into a plurality of tokens;

[0154] S4: Input the word unit into the target student network model, and perform inference calculation on the word unit based on the weight matrix including the zero vector column in the target student network model to determine the output result corresponding to the text data.

[0155] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0156] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in the natural language processing method embodiment are implemented, including:

[0157] S1: Get the pre-built teacher network model;

[0158] S2: Based on the teacher network model and the knowledge distillation mechanism, a target student network model is generated. The target student network model includes at least a weight matrix, and the weight matrix includes at least one zero vector column.

[0159] S3: In response to receiving the natural language processing request, obtaining text data and dividing the text data into a plurality of tokens;

[0160] S4: Input the word unit into the target student network model, and perform inference calculation on the word unit based on the weight matrix including the zero vector column in the target student network model to determine the output result corresponding to the text data.

[0161] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps in the natural language processing method embodiment are implemented, including:

[0162] S1: Get the pre-built teacher network model;

[0163] S2: Based on the teacher network model and the knowledge distillation mechanism, a target student network model is generated. The target student network model includes at least a weight matrix, and the weight matrix includes at least one zero vector column.

[0164] S3: In response to receiving the natural language processing request, obtaining text data and dividing the text data into a plurality of tokens;

[0165] S4: Input the word unit into the target student network model, and perform inference calculation on the word unit based on the weight matrix including the zero vector column in the target student network model to determine the output result corresponding to the text data.

[0166] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0167] The above is a detailed introduction to a natural language processing method, device, electronic device and storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A natural language processing method, characterized in that: The method comprises: Get a pre-built teacher network model; Based on the teacher network model and the knowledge distillation mechanism, a target student network model is generated, wherein the target student network model includes at least a weight matrix, and the weight matrix includes at least a zero vector column; In response to receiving a natural language processing request, obtaining text data and dividing the text data into a plurality of word units; Inputting the word-unit into the target student network model, and performing inference calculation on the word-unit based on the weight matrix including the zero vector column in the target student network model to determine an output result corresponding to the text data; Generating a target student network model based on the teacher network model and the knowledge distillation mechanism includes: Based on the teacher network model, a first student network model is constructed, wherein W′1 represents a weight matrix in the first student network model, W′1=W″1Q, Q represents a first diagonal matrix, and W″1 represents a randomly initialized matrix; based on the teacher network model and the first student network model, a training loss function is constructed, wherein the training loss function is a function with a distillation loss function and a task-related loss function as independent variables; Get training data; Based on the training loss function and the training data, training the first student network model to update the weight matrix in the first student network model; In response to the current training cycle being the preset training cycle, the training is ended to obtain a second student network model, wherein W′2 is used to represent a weight matrix in the second student network model, W′2=W″2Q, and W″2 represents a randomly initialized matrix after training update; Sort the diagonal elements in the first diagonal matrix in order of size; According to a preset ratio, multiple element values ​​are selected from the sorting results, and other element values ​​that are not selected are set to zero to generate a second diagonal matrix Q′; Based on the product of the second diagonal matrix Q′ and W″2, the weight matrix in the third student network model is determined to be W3=W″2Q′; Fixing the zero vector column in the weight matrix of the third student network model and setting the element value states in the remaining columns to a learnable state; Training the third student network model based on the task-related loss function and the text data corresponding to the target task node to update the element values ​​in the remaining columns; In response to the completion of the training, the target student network model is generated.

2. The natural language processing method according to claim 1, wherein: After generating the target student network model, the method further includes: Deploy the target student network model to the target task node.

3. The natural language processing method according to claim 1, wherein: Training the first student network model based on the training loss function and the training data to update the weight matrix in the first student network model includes: Fixing the weight matrix parameters of the teacher network model, randomly initializing the element values ​​in the first student network model, and other parameters; Set the parameters related to the stochastic gradient descent algorithm, define the current training cycle as e, the total training cycle as E, and in response to e≤E, perform the following operations: Add one to the current training period e; Randomly shuffling the order of samples in the training data and selecting a target batch of training samples from the training data; performing a training operation based on the training samples and parameters related to the stochastic gradient descent algorithm, calculating a training loss value, and updating a weight matrix in the first student network model using the stochastic gradient descent algorithm; In response to all training samples in the training data being utilized, ending the current training cycle; In response to the current training cycle being the preset training cycle, the training is ended to obtain the second student network model.

4. The natural language processing method according to claim 3, wherein: Randomly initializing the element values ​​in the first student network model includes: Based on Gaussian distribution, the element values ​​in the first student network model are randomly initialized.

5. The natural language processing method according to claim 1, wherein: Constructing a training loss function involves: The training loss function is constructed based on the distillation loss function and the task-related loss function.

6. The natural language processing method according to claim 5, characterized in that: The training loss function includes: L total =L task +βL M′，M ; Among them, L total Represents the training loss value, L task represents the task-related loss value, β represents the constant parameter, L M′，M Represents the distillation loss value.

7. The natural language processing method according to claim 5, characterized in that: The distillation loss function includes: W′ i =W″ i Q i ; Among them, L M′,M Represents the distillation loss value, N W represents the number of weight matrices in the teacher network model, L(W′ i ,W i ) represents the matrix W′ i and matrix W i Distillation loss value between i represents a random matrix, Q i represents a diagonal matrix.

8. The natural language processing method according to claim 7, characterized in that: The matrix W′ i and matrix W i The calculation method of the distillation loss value between includes: Where L1 represents the L1 norm, |*| F represents the norm of matrix F, γ represents the constant parameter, A vector representing the diagonal elements of a diagonal matrix.

9. The natural language processing method according to claim 5, characterized in that: The task-related loss functions include: Among them, L task Represents the training loss value, n represents the number of words, y ij Represents vector y i The jth element of Represents a vector The jth element of y i Represents a V-dimensional unique hot vector, indicating the category corresponding to the real word at position i, Represents the predicted V-dimensional probability distribution vector.

10. The natural language processing method according to claim 1, wherein: In response to receiving a natural language processing request, obtaining text data, and dividing the text data into a plurality of word units includes: Receive natural language processing requests from target task nodes; In response to receiving the natural language processing request, parsing the natural language processing request to obtain text data corresponding to the natural language processing request; The text data is divided according to a division rule to obtain a plurality of word units corresponding to the text data.

11. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the natural language processing method according to any one of claims 1 to 10 when executing the computer program.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the natural language processing method according to any one of claims 1 to 10 are implemented.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the natural language processing method according to any one of claims 1 to 10 are implemented.

Images