Text classification method, related device, equipment and storage medium

Abstract

The invention discloses a text classification method, a related device, equipment and a storage medium, and the method comprises the steps: carrying out the prediction based on a target classification task of a to-be-classified text, and obtaining an index value of a low-rank expert; intercepting at least part of the low-rank expert matrix based on the index value to obtain a target expert matrix; obtaining a first coded representation obtained by processing the text feature representation of the to-be-classified text by the pre-training language model, and obtaining a second coded representation obtained by processing the text feature representation by the target expert matrix; and performing prediction based on the first coded representation and the second coded representation to obtain a target category of the to-be-classified text under the target classification task. According to the scheme, under the scene of multiple text classification tasks, the accuracy of performing text classification by using the LoRA-assisted pre-training language model can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a text classification method and related devices, equipment and storage media. Background Art

[0002] With the rapid development of pre-trained language models, the model scale of pre-trained language models has gradually expanded, which has led to an increase in the fine-tuning cost.

[0003] In text classification tasks, the goal of efficient fine-tuning is to quickly improve the classification ability of the model with fewer parameter adjustments. Low-Rank Adaptation (LoRA) is a representative efficient fine-tuning method. However, when dealing with different text classification tasks, the existing LoRA is often inaccurate. In view of this, how to improve the accuracy of text classification using LoRA-assisted pre-trained language models in the scenario of multiple text classification tasks has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by this application is to provide a text classification method and related devices, equipment and storage media, which can improve the accuracy of text classification using the LoRA-assisted pre-trained language model in the scenario of multiple text classification tasks.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a text classification method, including: making predictions based on a target classification task of the text to be classified to obtain index values ​​of low-rank experts; intercepting at least part of the low-rank expert matrix based on the index values ​​to obtain a target expert matrix; obtaining a first encoded representation of the text feature representation of the text to be classified obtained by processing a pre-trained language model, and obtaining a second encoded representation of the text feature representation obtained by processing the target expert matrix; making predictions based on the first encoded representation and the second encoded representation to obtain a target category of the text to be classified under the target classification task.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a text classification device, including: an index prediction module, a matrix interception module, a feature encoding module and a category prediction module, the index prediction module is used to predict based on the target classification task of the text to be classified, and obtain the index value of the low-rank expert; the matrix interception module is used to intercept at least part of the low-rank expert matrix based on the index value to obtain the target expert matrix; the feature encoding module is used to obtain the text feature representation of the text to be classified by a first encoding representation obtained by processing a pre-trained language model, and obtain the text feature representation by The second encoding representation is processed by the target expert matrix; the category prediction module is used to predict based on the first encoding representation and the second encoding representation to obtain the target category of the text to be classified under the target classification task.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, which at least includes a memory and a processor coupled to each other, the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the text classification method in the above first aspect.

[0008] In order to solve the above technical problem, the fourth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the text classification method of the first aspect.

[0009] The above scheme predicts based on the target classification task of the text to be classified, obtains the index value of the low-rank expert, and intercepts at least part of the low-rank expert matrix based on the index value to obtain the target expert matrix, and then obtains the first encoding representation obtained by processing the text feature representation of the text to be classified by the pre-trained language model, and obtains the second encoding representation obtained by processing the text feature representation by the target expert matrix, so as to predict based on the first encoding representation and the second encoding representation to obtain the target category of the text to be classified under the target classification task, so that the index value of the expert matrix can be predicted according to the target classification task of the text to be classified, so as to adaptively select at least part of the low-rank expert matrix as the target expert matrix. Compared with the use of a fixed rank, this dynamic adjustment enables the pre-trained language model to efficiently handle multi-task scenarios, and avoids the accuracy drop caused by the difference in requirements between text classification tasks as much as possible, which helps to improve the adaptability of the pre-trained language model in multi-task scenarios. Therefore, in the scenario of multiple text classification tasks, the accuracy of text classification using the LoRA-assisted pre-trained language model can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of an embodiment of the text classification method of the present application;

[0011] Figure 2a It is a process diagram of an embodiment of the text classification method of the present application;

[0012] Figure 2b It is a visualization diagram of task embedding representation;

[0013] Figure 2c It is a schematic diagram of low-rank mixed expert rank allocation;

[0014] Figure 3 It is a schematic diagram of the framework of an embodiment of a text classification device of the present application;

[0015] Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0016] Figure 5It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0017] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0018] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0019] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0020] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of the text classification method of the present application.

[0021] Specifically, the following steps may be included:

[0022] Step S11: Make predictions based on the target classification task of the text to be classified to obtain the index value of the low-rank expert.

[0023] In an implementation scenario, the target classification task belongs to one of several preset classification tasks. Exemplarily, several preset classification tasks may include, but are not limited to: sentiment classification, natural language reasoning, grammar judgment, etc., and the specific connotations of several preset classification tasks will not be given one by one here. As a possible example, the target classification of the text to be classified is considered "sentiment classification", and its specific content is "Today's weather is so good!". Of course, the above example is only a possible example of the text to be classified in the actual application process. The specific content of the text to be classified and its target classification task are not limited here, and no examples are given one by one.

[0024] In one implementation scenario, as a possible implementation method, task embedding representations of several preset classification tasks can be obtained, and then predictions can be made based on the task embedding representations of the target classification task to obtain a prediction expert matrix, and the prediction expert matrix is ​​a one-dimensional matrix with the same length as the total number of tasks of the several preset classification tasks, so that the index value can be determined based on the maximum element in the prediction expert matrix. The above method obtains the prediction expert matrix through the task embedding representation of the target classification task, and predicts the maximum element in the expert matrix based on it to determine the index value, which can reduce the complexity of realizing adaptive selection of low-rank experts.

[0025] In a specific implementation scenario, the task embedding representation can be expressed as a real number vector of feature dimension h. For ease of description, the task embedding representation of the tth preset classification task can be recorded as e t ∈R h In addition, the task embedding representation of the preset classification task can be obtained through training and adjustment in the joint training process of the pre-trained language model and the low-rank expert matrix. For details, please refer to the following description of the joint training, which will not be repeated here.

[0026] In a specific implementation scenario, the prediction expert matrix can be predicted by a low-rank expert selection model, which can include weight parameters and bias parameters, and the prediction expert matrix can be obtained by summing the product of the weight parameters and the task embedding representation and the bias parameters. For ease of description, taking the t-th preset classification task as an example, its prediction expert matrix can be expressed as W g e t +b g , where W g represents the weight parameter of the low-rank expert selection model, b g represents the bias parameter of the low-rank expert selection model. On this basis, the index value of the largest element in the prediction expert matrix can be selected as the index value of the low-rank expert. For the sake of convenience, the t-th preset classification task is still taken as an example. The index value r of the low-rank expert corresponding to the t-th preset classification task is t It can be expressed as:

[0027] r t = argmax(W g e t +b g )……(1)

[0028] In the above formula (1), argmax represents the index value of the maximum element in the prediction expert matrix, and W g ∈R h×N , where N represents the total number of tasks of several preset classification tasks. In the above method, the prediction expert matrix is ​​predicted by the low-rank expert selection model, which includes weight parameters and bias parameters, and the prediction expert matrix is ​​obtained by summing the product of the weight parameters and the task embedding representation and the bias parameters. The index value of the low-rank expert can be determined by the low-rank expert selection model, which helps to improve the efficiency of determining the index value.

[0029] In another implementation scenario, different from the aforementioned implementation, as another possible implementation, the index values ​​corresponding to different preset classification tasks can also be pre-set, so that the index value corresponding to the target classification task can be directly determined based on this. Exemplarily, still taking several preset classification tasks including sentiment classification, natural language reasoning, and grammar judgment as an example, the index value corresponding to the preset classification task "sentiment classification" can be pre-set to 3, the index value corresponding to the preset classification task "natural language reasoning" can be pre-set to 5, and the index value corresponding to the preset classification task "grammar judgment" can be pre-set to 7. For the aforementioned text to be classified "Today's weather is so good!", since its target classification task is "sentiment classification", its index value can be directly determined to be 3. Of course, the above example is only a possible example in the actual application process, and other preset methods are not limited here, and no examples are given one by one.

[0030] Step S12: intercept at least part of the low-rank expert matrix based on the index value to obtain a target expert matrix.

[0031] Specifically, as mentioned above, the target classification task is one of several preset classification tasks. In addition, the low-rank expert matrix may include a dimension-raising matrix and a dimension-reducing matrix, and the number of rows of the dimension-raising matrix is ​​the same as the total number of tasks of the several preset classification tasks, and the number of columns of the dimension-reducing matrix is ​​the same as the total number of tasks of the several preset classification tasks. For the convenience of description, the dimension-raising matrix can be recorded as A∈R N×d , the dimension reduction matrix can be recorded as B∈R m×N , where d represents the output dimension of the pre-trained language model and m represents the input dimension of the pre-trained language model. On this basis, the first index value row in the dimension-increasing matrix can be intercepted to obtain the dimension-increasing expert matrix, and the first index value column in the dimension-reducing matrix can be intercepted to obtain the dimension-reducing expert matrix. Taking the tth preset classification task as an example, since its index value can be recorded as r t , so we can intercept the first r in the dimension-raising matrix A t row, and get the dimension-enhanced expert matrix And we can intercept the first r in the dimension reduction matrix B t Column, get the dimension reduction expert matrix Therefore, the target expert matrix can be obtained based on the dimension-up expert matrix and the dimension-down expert matrix. Specifically, the target expert matrix can be the product of the dimension-down expert matrix and the dimension-up expert matrix. Taking the t-th preset classification task as an example, the target expert matrix can be expressed as B t A t. Of course, the above example is only a possible example of the target expert matrix when taking the t-th preset classification task as an example. The specific representation of the target expert matrix under other possible situations will not be given one by one. In the above method, the target classification task is one of several preset classification tasks, and the low-rank expert matrix includes a dimension-raising matrix and a dimension-reducing matrix. The number of rows of the dimension-raising matrix is ​​the same as the total number of tasks of the several preset classification tasks, and the number of columns of the dimension-reducing matrix is ​​the same as the total number of tasks. The first index value row in the dimension-raising matrix is ​​intercepted to obtain the dimension-raising expert matrix, and the first index value column in the dimension-reducing matrix is ​​intercepted to obtain the dimension-reducing expert matrix. Based on the dimension-raising expert matrix and the dimension-reducing expert matrix, the target expert matrix is ​​obtained, and at least part of the low-rank expert matrix can be adaptively intercepted according to the index value to obtain the target expert matrix.

[0032] Step S13: obtaining a first encoding representation of the text feature representation of the text to be classified obtained by processing the pre-trained language model, and obtaining a second encoding representation of the text feature representation obtained by processing the target expert matrix.

[0033] For the sake of convenience, we still take the t-th preset classification task as an example. As mentioned above, the target expert matrix can be expressed as B t A t , then the text feature representation of the text to be classified is processed by the target expert matrix and the second encoding representation can be expressed as in, It represents the text feature representation of the i-th text to be classified under the t-th preset classification task. In addition, the text feature representation of the text to be classified can be obtained through embedding, word vector, etc., and the method for obtaining the text feature representation of the text to be classified is not limited here.

[0034] In one implementation scenario, as a possible implementation method, the text encoding representation output by the pre-trained language model after the text feature representation of the text to be classified is processed as a whole (that is, after each network layer in the pre-trained language model is processed in sequence) can be obtained as the first encoding representation. In other words, the low-rank expert can be set in the overall network model of the pre-trained language model.

[0035] In another implementation scenario, different from the aforementioned implementation, as another possible implementation, the low-rank expert can be set in at least one target network layer in the pre-trained language model, and each target network layer can be provided with a low-rank expert matrix. In this case, during the operation process of inputting the text feature representation of the text to be classified into the pre-trained language model, it can be detected whether the pre-trained language model runs to the target network layer, and in response to the pre-trained language model currently running to the target network layer, the aforementioned target classification task based on the text to be classified can be performed to predict and obtain the index value of the low-rank expert, until the output encoding representation obtained by processing the text feature representation by the target network layer is obtained as the first encoding representation, and the second encoding representation obtained by processing the text feature representation by the target expert matrix of the target network layer is obtained. It should be noted that in this process, the first index value row in the dimension-raising matrix contained in the low-rank expert matrix of the target network layer currently being run can be intercepted to obtain the dimension-raising expert matrix of the target network layer, and the first index value column in the dimension-reducing matrix contained in the low-rank expert matrix of the target network layer currently being run can be intercepted to obtain the dimension-reducing expert matrix of the target network layer, so that the target expert matrix of the target network layer currently being run can be obtained based on the dimension-raising expert matrix and the dimension-reducing expert matrix of the target network layer currently being run. For details, please refer to the above-mentioned related description, which will not be repeated here. On this basis, the first encoding representation and the second encoding representation can be fused as a new text feature representation, and the next network layer of the pre-trained language model can continue to process until the final encoding representation output by the last network layer of the pre-trained language model is obtained. That is to say, when the next network layer is not the target network layer, the next network layer can only process the new text feature representation to obtain the encoding feature representation output by the next network layer as the new text feature representation, which is then processed by the next network layer of the pre-trained language model; and when the next network layer is also the target network layer, the aforementioned index value prediction, encoding representation fusion and other operations can be continued to obtain the new text feature representation, which is then processed by the next network layer of the pre-trained language model, and this cycle is repeated until the last network layer of the pre-trained language model outputs the final encoding representation. In the above method, the low-rank expert is set in at least one target network layer in the pre-trained language model, and each target network layer is provided with a low-rank expert matrix. Then, when the pre-trained language model currently runs to the target network layer, a series of operations such as index value prediction and encoding representation fusion are performed to obtain the new text feature representation, which is then processed by the next network layer of the pre-trained language model until the final encoding representation output by the last network layer of the pre-trained language model is obtained. This method can use the network layer as the minimum auxiliary dimension of the low-rank expert in the pre-trained language model, which helps to improve the scalability and generalization of the low-rank hybrid expert model.

[0036] In a specific implementation scenario, the first encoding representation and the second encoding representation can be fused by methods such as addition, weighting, averaging, etc. to obtain a new text feature representation. The fusion method of the first encoding representation and the second encoding representation is not limited here.

[0037] In a specific implementation scenario, please refer to Figure 2a , Figure 2a Schematic diagram of the process of an embodiment of the text classification method of the present application. Figure 2a As shown, there can be an expert model with the same total number of tasks T as the preset classification tasks (such as Figure 2a The dotted rectangle in the middle shows expert 1, expert 2, expert 3, ..., expert T), and each expert model shares a part of the weight parameters of the preset language model, and the other part is the expert matrix, where the t-th expert matrix contains the dimension-upgraded expert matrix A T and the dimension reduction expert matrix B t , both of which are obtained by intercepting the first index value r in the dimension-raising matrix contained in the low-rank expert matrix t The first index value r in the row and the dimension reduction matrix contained in the low-rank expert matrix t The column is obtained, and the index value r t The task embedding representation of the t-th preset classification task is predicted by the router (i.e., the aforementioned low-rank expert selection model). In addition, as a possible example, the pre-trained language model can at least include a Transformer structure (such as Figure 2a In other words, when Figure 2a When the pre-trained language model runs to the attention layer, the expert model can be used to assist through the above process steps (such as Figure 2a The middle attention layer is extended to the middle part of the dotted rectangle through the dotted line), and when Figure 2a When the pre-trained language model runs to the feed-forward layer, the expert model can be used to assist through the above process steps (such as Figure 2a The feedforward layer in the middle is extended to the dotted rectangular box in the middle part through the dotted line). Of course, Figure 2a What is shown is only a possible example in actual application. The network structure of the pre-trained language model is not limited here, and the specific settings of the target network layer are not limited.

[0038] Step S14: Predicting based on the first encoding representation and the second encoding representation to obtain a target category of the text to be classified under the target classification task.

[0039] Specifically, as mentioned above, as a possible example, the low-rank expert can be set in the overall network model of the pre-trained language model, then the first encoding representation and the second encoding representation can be fused (e.g., weighted, averaged, added, etc.) to obtain the final encoding representation, and then the classification prediction is performed through the category prediction model (e.g., it can include a fully connected layer, softmax, etc.) to obtain the target category of the text to be classified under the target classification task. For the sake of ease of description, still taking the i-th text to be classified under the t-th preset classification task as an example, its target category It can be expressed as:

[0040]

[0041] In the above formula (2), represents the first encoding representation of the text feature representation of the i-th text to be classified under the t-th preset classification task after being processed by the pre-trained language model, represents the second encoding representation of the text feature representation of the i-th text to be classified under the t-th preset classification task, which is obtained by processing the target expert matrix, and C represents the category prediction model. Alternatively, as another possible example, the low-rank expert can also be set in at least one target network layer in the pre-trained language model. Then, as mentioned above, when the pre-trained language model runs to the target network layer, a new text feature representation can be obtained through a series of operations such as index value prediction and encoding representation fusion, and the next network layer of the pre-trained language model continues to process until the final encoding representation output by the last network layer of the pre-trained language model is obtained. At this time, prediction can be made based on the final encoding representation to obtain the target category of the text to be classified under the target classification task. For ease of understanding, taking the aforementioned text to be classified "Today's weather is so good!" as an example, as mentioned above, its target classification task is "emotion classification", then through the aforementioned process steps, its target category can be obtained as "pleasure". Of course, the above example is only a possible example in the actual application process, and the case where the target classification task is other preset classification tasks or the text to be classified is other text content will not be given one by one.

[0042] In addition, as mentioned above, at least the low-rank expert matrix is ​​obtained by joint training with the pre-trained language model based on sample data, and the fixed parameters of the pre-trained language model remain unchanged during the joint training process (such as Figure 2a The sample data includes sample texts of several preset classification tasks, and the sample texts are annotated with the real categories under the preset classification tasks. Taking the total number of tasks N of the preset classification tasks as an example, we can construct a text classification task T = {T1, T2, …, T t ,…,T N}, where T trepresents the t-th preset classification task, then the sample text set of the t-th preset classification task can be recorded as in, represents the i-th sample text of the t-th preset classification task, Represents sample text The labeled true category under the preset classification task, |D t | represents the total number of sample texts in the sample text set of the tth preset classification task. On this basis, prediction can be made based on the preset classification task to which the sample text belongs to obtain the prediction index value of the low-rank expert (for details, please refer to the relevant description of obtaining the index value mentioned above), and at least part of the low-rank expert matrix can be intercepted based on the prediction index value to obtain the prediction expert matrix (for details, please refer to the relevant description of obtaining the target expert matrix mentioned above), so as to obtain the first sample encoding representation obtained by processing the sample text feature representation of the sample text by the pre-trained language model (for details, please refer to the relevant description of obtaining the first encoding representation mentioned above), and obtain the second sample encoding representation obtained by processing the sample text feature representation by the prediction expert matrix (for details, please refer to the relevant description of obtaining the second encoding representation mentioned above), and then make a prediction based on the first sample encoding representation and the second sample encoding representation to obtain the predicted category of the sample text under the preset classification task to which it belongs (for details, please refer to the relevant description of obtaining the target category mentioned above), so as to at least adjust the low-rank expert matrix based on the difference between the true category and the predicted category of the sample text under the preset classification task to which it belongs, so as to force the predicted category to converge to the true category as much as possible, so that the predicted text category can be learned as much as possible during the training process, and since there is no need to adjust the pre-trained language model, the training parameters and computing resource consumption can be reduced as much as possible. Furthermore, as mentioned above, prediction can be made based on the task embedding representation of the preset classification task to which the sample text belongs, and the predicted index value of the low-rank expert can be obtained (for details, please refer to the above description of obtaining the index value). In order to assist in training, the contrast loss can be obtained based on the first similarity between the task embedding representation of the preset classification task to which the sample text belongs and the first sample encoding representation, and the second similarity between the task embedding representation of each preset classification task and the first sample encoding representation. The contrast loss is negatively correlated with the first similarity and positively correlated with the second similarity. The generation loss is obtained based on the difference between the true category and the predicted category of the sample text under the preset classification task. Then, based on the generation loss and the contrast loss, at least the low-rank expert matrix and the task embedding representation can be adjusted. Of course, as mentioned above, the index value prediction can be obtained by processing the low-rank expert selection model. In the process of adjusting the low-rank expert matrix and the task embedding representation, the network parameters of the low-rank expert selection model can also be adjusted together. Exemplarily, the generation loss L gen It can be measured by the cross entropy loss function, which can be expressed as:

[0043]

[0044] In the above formula (3), N represents the total number of tasks of several preset classification tasks, |D t | represents the total number of sample texts in the sample data set of the t-th preset classification task, represents the predicted category obtained by processing the i-th sample text under the t-th preset classification task, represents the true category of the i-th sample text under the t-th preset classification task. In addition, the contrast loss L con It can be measured by contrastive learning loss function, which can be expressed as:

[0045]

[0046] In the above formula (4), τ represents the temperature coefficient, sim represents the similarity calculation, and e k represents the task embedding representation of the k-th preset classification task, represents the first sample encoding representation of the i-th sample text under the t-th preset classification task. On this basis, the generation loss L gen and contrast loss L con Weighted, we get the training loss L:

[0047] L=L gen +λL con ……(5)

[0048] In the above formula (5), λ represents the weighting coefficient. After obtaining the training loss L, as a possible example, the AdamW optimizer can be used to train the low-rank mixed expert model and update the relevant parameters until the maximum number of iterations is reached or the training loss L converges. The above method, by contrasting the learning optimization and balanced data sampling strategy for the text classification task embedding, pays special attention to the performance of small data set tasks. After the training is completed, further testing is performed on the test set, and the experimental results shown in Table 1 can be obtained.

[0049] Table 1 Experimental results of the technical solution of the embodiment of the present disclosure on the glue dataset

[0050]

[0051] As shown in Table 1, Finetuning represents a full-scale fine-tuning method, Adapters represents an adapter-based method, PT represents a prompt-learning-based method, LoRA is a low-rank adaptation method, HyperFomer is an adapter-based hypernetwork method, MPT is a multi-task migration method based on prompt learning, multiLoRA and MixLoRA are parallel LoRA methods, and MoRE is a method proposed in the embodiment of the present disclosure. In addition, MNLI is a natural language reasoning task dataset, QQP is a semantic repetition detection task dataset, QNLI is a semantic implication task dataset, SST-2 is a sentiment classification task dataset, STS-B is a semantic similarity task dataset, MRPC is a synonym judgment task dataset, RTE is a text implication recognition task dataset, and COLA is a grammatical judgment task dataset. In addition, migration experiments are also conducted in low-resource scenarios, and the experimental results are shown in Table 2. Among them, k-shot represents the number of training samples in low-resource scenarios. BoolQ is a binary question-answering task dataset, CB is a natural language reasoning task dataset, and SciTail is a text implication task dataset.

[0052] Table 2 Experimental results of the technical solution of the embodiment of the present disclosure in a low-resource scenario

[0053]

[0054] As can be seen from Table 1 and Table 2, the embodiments of the present disclosure improve the average performance of the pre-trained language model on the test set, especially improve its performance on a small data set with a relatively small training set. In low-resource scenarios, the embodiments of the present disclosure also achieve the best migration performance. These experimental results fully demonstrate the effectiveness and practicality of the present invention. Figure 2b and Figure 2c , Figure 2b It is a visualization diagram of the task embedding representation. Figure 2c It is a schematic diagram of low-rank mixed expert rank allocation. Figure 2b The visualization of the task embedding representation designed by the embodiment of the present disclosure is shown, and the visualization of the task embedding representation in the query, key, and value of the pre-trained language model is shown respectively. It can be observed that after training, the model can learn the task embedding representation of similar text classification tasks. For example, the MNLI task and the MRPC task are always more similar. Figure 2c It shows the low-rank expert allocation under different tasks (the horizontal axis represents the expert allocation under different tasks, and the vertical axis represents the distribution count). Expert1 means that the task is assigned to the expert matrix with rank 1, expert2 means that the task is assigned to the expert matrix with rank 2, and so on. Figure 2cIt can be seen that different text classification tasks have different requirements for ranks. For example, the CoLA task tends to be assigned to an expert matrix with a rank of 2, while the MNLI task tends to be assigned to an expert matrix with a rank of 1. The appropriate ranks are adaptively assigned to different tasks, which is also consistent with the design intention of the technical solution disclosed in the embodiments of the present disclosure.

[0055] The above scheme predicts based on the target classification task of the text to be classified, obtains the index value of the low-rank expert, and intercepts at least part of the low-rank expert matrix based on the index value to obtain the target expert matrix, and then obtains the first encoding representation obtained by processing the text feature representation of the text to be classified by the pre-trained language model, and obtains the second encoding representation obtained by processing the text feature representation by the target expert matrix, so as to predict based on the first encoding representation and the second encoding representation to obtain the target category of the text to be classified under the target classification task, so that the index value of the expert matrix can be predicted according to the target classification task of the text to be classified, so as to adaptively select at least part of the low-rank expert matrix as the target expert matrix. Compared with the use of a fixed rank, this dynamic adjustment enables the pre-trained language model to efficiently handle multi-task scenarios, and avoids the accuracy drop caused by the difference in requirements between text classification tasks as much as possible, which helps to improve the adaptability of the pre-trained language model in multi-task scenarios. Therefore, in the scenario of multiple text classification tasks, the accuracy of text classification using the LoRA-assisted pre-trained language model can be improved.

[0056] See also Figure 3 , Figure 3 It is a schematic diagram of the framework of an embodiment of the text classification device of the present application. The text classification device 30 includes: an index prediction module 31, a matrix interception module 32, a feature encoding module 33 and a category prediction module 34. The index prediction module 31 is used to make predictions based on the target classification task of the text to be classified to obtain the index value of the low-rank expert; the matrix interception module 32 is used to intercept at least part of the low-rank expert matrix based on the index value to obtain the target expert matrix; the feature encoding module 33 is used to obtain the first encoding representation obtained by processing the text feature representation of the text to be classified by the pre-trained language model, and obtain the second encoding representation obtained by processing the text feature representation by the target expert matrix; the category prediction module 34 is used to make predictions based on the first encoding representation and the second encoding representation to obtain the target category of the text to be classified under the target classification task.

[0057] In the above scheme, the text classification device 30 makes predictions based on the target classification task of the text to be classified, obtains the index value of the low-rank expert, and intercepts at least part of the low-rank expert matrix based on the index value to obtain the target expert matrix, and then obtains the first encoding representation obtained by processing the text feature representation of the text to be classified by the pre-trained language model, and obtains the second encoding representation obtained by processing the text feature representation by the target expert matrix, so as to make predictions based on the first encoding representation and the second encoding representation to obtain the target category of the text to be classified under the target classification task, so that the index value of the expert matrix can be predicted according to the target classification task of the text to be classified, so as to adaptively select at least part of the low-rank expert matrix as the target expert matrix. Compared with the use of a fixed rank, this dynamic adjustment enables the pre-trained language model to efficiently handle multi-task scenarios, and avoids the accuracy reduction caused by the difference in requirements between text classification tasks as much as possible, which helps to improve the adaptability of the pre-trained language model in multi-task scenarios. Therefore, in the scenario of multiple text classification tasks, the accuracy of text classification using the LoRA-assisted pre-trained language model can be improved.

[0058] In some disclosed embodiments, the index prediction module 31 includes an embedding submodule for obtaining task embedding representations of several preset classification tasks; the index prediction module 31 includes a prediction submodule for making predictions based on the task embedding representations of the target classification task to obtain a prediction expert matrix; wherein the prediction expert matrix is ​​a one-dimensional matrix with the same length as the total number of tasks of several preset classification tasks; the index prediction module 31 includes a determination submodule for determining an index value based on the maximum matrix element in the prediction expert matrix.

[0059] In some disclosed embodiments, the predicted expert matrix is ​​predicted by a low-rank expert selection model, which includes weight parameters and bias parameters, and the predicted expert matrix is ​​obtained by multiplying the weight parameters by the task embedding representation and summing the bias parameters.

[0060] In some disclosed embodiments, the target classification task is one of several preset classification tasks, the low-rank expert matrix includes an increased dimension matrix and a reduced dimension matrix, the number of rows of the increased dimension matrix is ​​the same as the total number of tasks of the several preset classification tasks, and the number of columns of the reduced dimension matrix is ​​the same as the total number of tasks, the matrix interception module 32 includes a first interception submodule, which is used to intercept the first index value row in the increased dimension matrix to obtain the increased dimension expert matrix; the matrix interception module 32 includes a second interception submodule, which is used to intercept the first index value column in the reduced dimension matrix to obtain the reduced dimension expert matrix; the matrix interception module 32 includes a matrix acquisition submodule, which is used to obtain the target expert matrix based on the increased dimension expert matrix and the reduced dimension expert matrix.

[0061] In some disclosed embodiments, the target expert matrix is ​​the product of the dimension reduction expert matrix and the dimension increase expert matrix.

[0062] In some disclosed embodiments, the low-rank expert is provided in at least one target network layer in the pre-trained language model, and each target network layer is provided with a low-rank expert matrix. The text classification device 30 includes a network detection module for detecting whether the pre-trained language model runs to the target network layer; the text classification device 30 includes a feature encoding module for performing a target classification task based on the text to be classified for prediction in response to the pre-trained language model currently running to the target network layer, and obtaining an index value of the low-rank expert until the output encoding representation obtained by processing the text feature representation by the target network layer is obtained as the first encoding representation, and the second encoding representation obtained by processing the text feature representation by the target expert matrix of the target network layer is obtained; the text classification device 30 includes a loop iteration module for fusing the first encoding representation and the second encoding representation as a new text feature representation, and continuing to process by the next network layer of the pre-trained language model until the final encoding representation output by the last network layer of the pre-trained language model is obtained; the text classification device 30 includes a target prediction module for predicting based on the final encoding representation to obtain the target category of the text to be classified under the target classification task.

[0063] In some disclosed embodiments, the pre-trained language model includes at least a Transformer structure, and the target network layer includes at least one of an attention layer and a feed-forward layer in the Transformer structure.

[0064] In some disclosed embodiments, at least the low-rank expert matrix is ​​obtained by joint training with a pre-trained language model based on sample data, the sample data includes sample texts of several preset classification tasks, the sample texts are annotated with the true categories under the corresponding preset classification tasks, and the fixed parameters of the pre-trained language model remain unchanged during the joint training process.

[0065] In some disclosed embodiments, the text classification device 30 includes a sample index prediction module, which is used to make predictions based on a preset classification task to which the sample text belongs, and obtain a prediction index value of a low-rank expert; the text classification device 30 includes a sample matrix prediction module, which is used to intercept at least a part of the low-rank expert matrix based on the prediction index value to obtain a prediction expert matrix; the text classification device 30 includes a sample feature encoding module, which is used to obtain a first sample encoding representation of a sample text feature representation of the sample text obtained by processing a pre-trained language model, and obtain a second sample encoding representation of the sample text feature representation obtained by processing a prediction expert matrix; the text classification device 30 includes a sample category prediction module, which is used to make predictions based on the first sample encoding representation and the second sample encoding representation to obtain a predicted category of the sample text under the preset classification task; the text classification device 30 includes an expert matrix adjustment module, which is used to at least adjust the low-rank expert matrix based on the difference between the true category and the predicted category of the sample text under the preset classification task.

[0066] In some disclosed embodiments, the sample index prediction module is specifically used to make predictions based on the task embedding representation of the preset classification task to which the sample text belongs, and obtain a prediction index value of a low-rank expert; the text classification device 30 also includes a contrast loss measurement module, which is used to obtain the contrast loss based on a first similarity between the task embedding representation of the preset classification task to which the sample text belongs and the first sample encoding representation, and a second similarity between the task embedding representation of each preset classification task and the first sample encoding representation; wherein the contrast loss is negatively correlated with the first similarity and positively correlated with the second similarity; the expert matrix adjustment module is specifically used to obtain the generation loss based on the difference between the true category and the predicted category of the sample text under the preset classification task to which it belongs, and based on the generation loss and the contrast loss, at least adjust the low-rank expert matrix and the task embedding representation.

[0067] See also Figure 4 , Figure 4 It is a schematic diagram of the framework of an embodiment of an electronic device of the present application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other, the memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-mentioned text classification method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 40 may include but is not limited to an office notebook, a smart phone, a tablet computer, a learning machine, etc., and the specific type of the electronic device 40 is not limited here.

[0068] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above-mentioned intelligent dialogue method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0069] In the above scheme, the electronic device 40 makes predictions based on the target classification task of the text to be classified, obtains the index value of the low-rank expert, and intercepts at least part of the low-rank expert matrix based on the index value to obtain the target expert matrix, and then obtains the first coding representation obtained by processing the text feature representation of the text to be classified by the pre-trained language model, and obtains the second coding representation obtained by processing the text feature representation by the target expert matrix, so as to make predictions based on the first coding representation and the second coding representation to obtain the target category of the text to be classified under the target classification task, so that the index value of the expert matrix can be predicted according to the target classification task of the text to be classified, so as to adaptively select at least part of the low-rank expert matrix as the target expert matrix. Compared with the use of a fixed rank, this dynamic adjustment enables the pre-trained language model to efficiently handle multi-task scenarios, and avoids the accuracy drop caused by the difference in requirements between text classification tasks as much as possible, which helps to improve the adaptability of the pre-trained language model in multi-task scenarios. Therefore, in the scenario of multiple text classification tasks, the accuracy of text classification using the LoRA-assisted pre-trained language model can be improved.

[0070] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 50 of the present application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor, and the program instructions 51 are used to implement the steps in any of the above-mentioned text classification method embodiments.

[0071] In the above scheme, the computer-readable storage medium 50 makes predictions based on the target classification task of the text to be classified, obtains the index value of the low-rank expert, and intercepts at least part of the low-rank expert matrix based on the index value to obtain the target expert matrix, and then obtains the first encoding representation obtained by processing the text feature representation of the text to be classified by the pre-trained language model, and obtains the second encoding representation obtained by processing the text feature representation by the target expert matrix, so as to make predictions based on the first encoding representation and the second encoding representation to obtain the target category of the text to be classified under the target classification task, so that the index value of the expert matrix can be predicted according to the target classification task of the text to be classified, so as to adaptively select at least part of the low-rank expert matrix as the target expert matrix. Compared with the use of a fixed rank, this dynamic adjustment enables the pre-trained language model to efficiently handle multi-task scenarios, and avoids the accuracy drop caused by the difference in requirements between text classification tasks as much as possible, which helps to improve the adaptability of the pre-trained language model in multi-task scenarios. Therefore, in the scenario of multiple text classification tasks, the accuracy of text classification using the LoRA-assisted pre-trained language model can be improved.

[0072] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0073] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0074] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0075] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0076] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0077] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0078] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A text classification method, characterized in that: include: Based on the target classification task of the text to be classified, prediction is performed to obtain the index value of the low-rank expert; Based on the index value, at least a portion of the low-rank expert matrix is ​​intercepted to obtain a target expert matrix; Acquire a first encoding representation of the text feature representation of the text to be classified obtained by processing the pre-trained language model, and acquire a second encoding representation of the text feature representation obtained by processing the target expert matrix; Prediction is performed based on the first encoded representation and the second encoded representation to obtain a target category of the text to be classified under the target classification task.

2. The method according to claim 1, characterized in that The target classification task based on the text to be classified is predicted to obtain the index value of the low-rank expert, including: Obtain task embedding representations for several preset classification tasks; Perform prediction based on the task embedding representation of the target classification task to obtain a prediction expert matrix; wherein the prediction expert matrix is ​​a one-dimensional matrix with the same length as the total number of tasks of the plurality of preset classification tasks; The index value is determined based on a maximum matrix element in the prediction expert matrix.

3. The method according to claim 2, characterized in that The prediction expert matrix is ​​predicted by a low-rank expert selection model, which includes a weight parameter and a bias parameter, and the prediction expert matrix is ​​obtained by summing the product of the weight parameter and the task embedding representation and the bias parameter.

4. The method according to claim 1, characterized in that: The target classification task is one of several preset classification tasks, the low-rank expert matrix includes a dimension-raising matrix and a dimension-reducing matrix, the number of rows of the dimension-raising matrix is ​​the same as the total number of tasks of the several preset classification tasks, the number of columns of the dimension-reducing matrix is ​​the same as the total number of tasks, and the target expert matrix is ​​obtained by intercepting at least part of the low-rank expert matrix based on the index value, including: The index value rows mentioned above are intercepted in the dimension-raising matrix to obtain the dimension-raising expert matrix, and the index value columns mentioned above are intercepted in the dimension-reducing matrix to obtain the dimension-reducing expert matrix; The target expert matrix is ​​obtained based on the dimension-raising expert matrix and the dimension-reducing expert matrix.

5. The method according to claim 4, characterized in that The target expert matrix is ​​the product of the dimension reduction expert matrix and the dimension increase expert matrix.

6. The method according to claim 1, characterized in that The low-rank expert is provided in at least one target network layer in the pre-trained language model, and each target network layer is provided with the low-rank expert matrix. Before the target classification task based on the text to be classified is predicted and the index value of the low-rank expert is obtained, the method comprises: Detecting whether the pre-trained language model runs to the target network layer; In response to the pre-trained language model currently running to the target network layer, the target classification task based on the text to be classified is performed for prediction to obtain the index value of the low-rank expert, until the output encoding representation obtained by processing the text feature representation by the target network layer is obtained as the first encoding representation, and the second encoding representation obtained by processing the text feature representation by the target expert matrix of the target network layer is obtained; Based on the first encoded representation and the second encoded representation, the first encoded representation and the second encoded representation are fused as a new text feature representation, and the next network layer of the pre-trained language model continues to process the new text feature representation until the final encoded representation output by the last network layer of the pre-trained language model is obtained; Prediction is performed based on the final encoded representation to obtain the target category of the text to be classified under the target classification task.

7. The method according to claim 6, characterized in that The pre-trained language model includes at least a Transformer structure, and the target network layer includes at least one of an attention layer and a feedforward layer in the Transformer structure.

8. The method according to claim 1, characterized in that At least the low-rank expert matrix is ​​obtained by joint training with the pre-trained language model based on sample data, wherein the sample data includes sample texts of several preset classification tasks, and the sample texts are annotated with the true categories under the preset classification tasks to which they belong, and the fixed parameters of the pre-trained language model remain unchanged during the joint training process.

9. The method according to claim 8, characterized in that The joint training includes: Perform prediction based on the preset classification task to which the sample text belongs, and obtain a prediction index value of a low-rank expert; Based on the prediction index value, at least a portion of the low-rank expert matrix is ​​intercepted to obtain a prediction expert matrix; Acquire a first sample encoding representation of the sample text feature representation of the sample text obtained by processing the pre-trained language model, and acquire a second sample encoding representation of the sample text feature representation obtained by processing the prediction expert matrix; Performing prediction based on the first sample encoding representation and the second sample encoding representation to obtain a predicted category of the sample text under the preset classification task to which it belongs; Based on the difference between the true category and the predicted category of the sample text under the preset classification task, at least the low-rank expert matrix is ​​adjusted.

10. The method according to claim 9, characterized in that The step of performing prediction based on the preset classification task to which the sample text belongs to obtain a prediction index value of a low-rank expert includes: Perform prediction based on the task embedding representation of the preset classification task to which the sample text belongs, to obtain a prediction index value of a low-rank expert; Before at least adjusting the low-rank expert matrix based on the difference between the true category and the predicted category of the sample text under the preset classification task, the method includes: Based on the first similarity between the task embedding representation of the preset classification task to which the sample text belongs and the first sample encoding representation, and the second similarity between the task embedding representations of each of the preset classification tasks and the first sample encoding representation, a contrast loss is obtained; wherein the contrast loss is negatively correlated with the first similarity and positively correlated with the second similarity; The step of at least adjusting the low-rank expert matrix based on the difference between the true category and the predicted category of the sample text under the preset classification task includes: Based on the difference between the true category and the predicted category of the sample text under the preset classification task, a generation loss is obtained, and based on the generation loss and the contrast loss, at least the low-rank expert matrix and the task embedding representation are adjusted.

11. A text classification device, characterized in that: include: The index prediction module is used to predict the target classification task based on the text to be classified and obtain the index value of the low-rank expert; A matrix interception module, used for intercepting at least part of the low-rank expert matrix based on the index value to obtain a target expert matrix; A feature encoding module, used for obtaining a first encoding representation of the text feature representation of the to-be-classified text by processing the pre-trained language model, and obtaining a second encoding representation of the text feature representation by processing the target expert matrix; A category prediction module is used to predict based on the first encoding representation and the second encoding representation to obtain a target category of the text to be classified under the target classification task.

12. An electronic device, characterized in that: The invention at least comprises a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the text classification method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the text classification method according to any one of claims 1 to 10.

Images