Multi-level label generation model training method and device and text classification method and device

Abstract

The invention discloses a training method and device of a multi-level label generation model and a text classification method and device. The training method of the multi-level label generation model comprises the following steps: acquiring a plurality of training samples; the training samples are input into an original model, the original model comprises a sequence-to-sequence model and a multi-label classifier, and the input of the multi-label classifier is the output of an encoder in the sequence-to-sequence model; and training to obtain a multi-level label generation model according to the difference between the non-level prediction label of each standard text output by the multi-label classifier and the matched non-level labeling label and the difference between the multi-level prediction label of each standard text output by the sequence-to-sequence model and the matched multi-level labeling label. According to the technical scheme provided by the embodiment of the invention, the potential relationship between the text and the tag to which the text belongs is mined to the greatest extent, and the prediction accuracy of the multi-level tag can be further improved.

Description

Technical Field

[0001] The present invention relates to the field of text processing technology, and in particular to a training and text classification method and device for a multi-level label generation model. Background Art

[0002] The task of text classification is to assign one or more predefined category labels to the text according to its content. With the increasing richness and complexity of network information, the task of text classification has gradually shifted from ordinary single-level label classification tasks to multi-level label classification tasks.

[0003] Among them, the multi-level label classification task can be understood as classifying text in a hierarchical label space, which is relatively difficult and has relatively complex application scenarios. Figure 1 A schematic diagram of a three-level label space is shown in FIG. Figure 1 As shown, the text classification label is multi-level. The first-level label contains one or more second-level labels, and the second-level label contains one or more third-level labels. After completing the multi-level label classification for a text, you can get a multi-level label in the form of: "(first-level label: science, education, culture and tourism)-(second-level label: science and technology life)-(third-level label: other scientific and technological information)".

[0004] However, existing multi-level label classification technologies often have problems such as low classification accuracy and classification results that do not meet the requirements of the label hierarchy architecture. Therefore, how to provide an efficient and accurate multi-level label classification method is an important issue that needs to be solved. Summary of the invention

[0005] The embodiment of the present invention provides a training method and device for a multi-level label generation model, so as to realize efficient and accurate prediction of multi-level labels for text.

[0006] According to one aspect of an embodiment of the present invention, a method for training a multi-level label generation model is provided, the method comprising:

[0007] Acquire multiple training samples, the training samples include standard text, multi-level annotation labels of the standard text, and non-level annotation labels of the standard text;

[0008] Input each training sample into the original model, where the original model includes a sequence-to-sequence model and a multi-label classifier, and the input of the multi-label classifier is the output of the encoder in the sequence-to-sequence model;

[0009] According to the difference between the non-hierarchical prediction label of each standard text output by the multi-label classifier and the matching non-hierarchical annotation label, and the difference between the multi-level prediction label of each standard text output by the sequence-to-sequence model and the matching multi-level annotation label, a multi-level label generation model is trained;

[0010] Among them, the output of the multi-level label generation model is the output of the decoder in the trained sequence-to-sequence model.

[0011] According to another aspect of the present invention, a text classification method is provided, the method comprising:

[0012] Get the target text to be classified;

[0013] Inputting the target text into a pre-trained multi-level label generation model to obtain a multi-level predicted label of the target text; wherein the multi-level label generation model is trained by the training method of the multi-level label generation model as described in any one of the embodiments of the present invention;

[0014] The target text is classified into a text set matching the multi-level predicted labels.

[0015] According to another aspect of an embodiment of the present invention, a training device for a multi-level label generation model is provided, the device comprising:

[0016] A training sample acquisition module is used to acquire multiple training samples, the training samples include standard text, multi-level annotation labels of the standard text and non-level annotation labels of the standard text;

[0017] A training sample input module, used to input each training sample into the original model, wherein the original model includes a sequence-to-sequence model and a multi-label classifier, and the input of the multi-label classifier is the output of the encoder in the sequence-to-sequence model;

[0018] A model training module, used for training a multi-level label generation model according to the difference between the non-hierarchical prediction label of each standard text output by the multi-label classifier and the matching non-hierarchical annotation label, and the difference between the multi-level prediction label of each standard text output by the sequence-to-sequence model and the matching multi-level annotation label;

[0019] Among them, the output of the multi-level label generation model is the output of the decoder in the trained sequence-to-sequence model.

[0020] According to another aspect of an embodiment of the present invention, a text classification device is also provided, the device comprising:

[0021] A target text acquisition module is used to acquire the target text to be classified;

[0022] A multi-level prediction label acquisition module, used to input the target text into a pre-trained multi-level label generation model to obtain the multi-level prediction labels of the target text; wherein the multi-level label generation model is trained by the training method of the multi-level label generation model as described in any one of the embodiments of the present invention;

[0023] The target text classification module is used to classify the target text into a text set that matches the multi-level prediction label.

[0024] According to another aspect of the present invention, there is also provided an electronic device, the electronic device comprising:

[0025] at least one processor; and

[0026] a memory communicatively connected to the at least one processor; wherein,

[0027] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the training method of the multi-level label generation model as described in any embodiment of the present invention, or execute the text classification method as described in any embodiment of the present invention.

[0028] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a training method for a multi-level label generation model as described in any embodiment of the present invention, or to implement a text classification method as described in any embodiment of the present invention when executed.

[0029] The technical solution of the embodiment of the present invention obtains multiple training samples and inputs them into the original model. According to the difference between the non-hierarchical prediction labels of each standard text output by the multi-label classifier in the original model and the matching non-hierarchical annotation labels, as well as the difference between the multi-level prediction labels of each standard text output by the sequence-to-sequence model in the original model and the matching multi-level annotation labels, a multi-level label generation model is trained, and a technical means for performing multi-level label prediction on the target text based on the multi-level label generation model is adopted. On the basis of the structure of the sequence-to-sequence model, the learning of multi-label classification tasks is integrated. On the basis of making full use of the rich semantic information and understanding ability in the sequence-to-sequence model, the potential relationship between the text and its corresponding label is mined to the greatest extent, which can further improve the prediction accuracy of the multi-level label.

[0030] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 It is a schematic diagram of a three-level label space in the prior art;

[0033] Figure 2 is a flowchart of a training method for a multi-level label generation model provided according to Embodiment 1 of the present invention;

[0034] Figure 3 is a flowchart of a training method for a multi-level label generation model provided according to Embodiment 2 of the present invention;

[0035] Figure 4 It is a structural schematic diagram of an original model applicable to the second embodiment of the present invention;

[0036] Figure 5 is a flowchart of a text classification method provided according to Embodiment 3 of the present invention;

[0037] Figure 6 is a flowchart of another text classification method provided according to Embodiment 4 of the present invention;

[0038] Figure 7 is a structural diagram of a training device for a multi-level label generation model provided according to Embodiment 5 of the present invention;

[0039] Figure 8 is a structural diagram of a text classification device provided according to Embodiment 6 of the present invention;

[0040] Fig. 9 It is a structural schematic diagram of an electronic device for implementing a training method for a multi-level label generation model or a text classification method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0043] Embodiment 1

[0044] Figure 2 This is a flowchart of a method for training a multi-level label generation model provided in the first embodiment of the present invention. This embodiment can be applied to the case where a multi-level label generation model is obtained through training, and the multi-level label generation model is used to generate multi-level labels for input text. The method can be executed by a training device for a multi-level label generation model, and the training device for the multi-level label generation model can be implemented in the form of hardware and / or software. The training device for the multi-level label generation model can be configured in an electronic device with data processing capabilities, such as a terminal or a server. Figure 2 As shown, the method includes:

[0045] S210 , obtaining a plurality of training samples, wherein the training samples include standard text, multi-level annotation labels of the standard text, and non-level annotation labels of the standard text.

[0046] In this embodiment, in order to finally train and obtain a multi-level label generation model, it is necessary to first obtain training samples for model training.

[0047] Each training sample includes: standard text, multi-level annotation labels corresponding to the standard text, and non-hierarchical annotation labels corresponding to the standard text. Multi-level annotation labels can be understood as a set of labels with a hierarchical structure, for example, "(first-level label: science, education, culture and tourism)-(second-level label: science and technology life)-(third-level label: other science and technology information)", and non-hierarchical annotation labels can be understood as a set of labels without a hierarchical structure, for example: "science, education, culture and tourism, science and technology life and other science and technology information", and the labels are in parallel relationship.

[0048] In an optional implementation of this embodiment, obtaining multiple training samples may include:

[0049] Capture information text as the current standard text in the information website, and obtain the search directory where the current standard text is located; organize the directory items of each level in the search directory according to a preset hierarchical label organization format to obtain a multi-level annotation label of the current standard text; determine all directory items included in the search directory as non-hierarchical annotation labels of the current standard text.

[0050] Optionally, in order to obtain a large number of correctly annotated training samples, it is possible to consider capturing text with a multi-level target structure from existing information websites, and based on the multi-level target structure, generate corresponding multi-level annotation labels and non-hierarchical annotation labels.

[0051] Specifically, the information website can be understood as a website that stores various types of text information (for example, news, papers or blogs, etc.). In the information website, in order to effectively retrieve information texts, a corresponding search directory can be set for each information text, for example: a search directory in the form of: "Basic Education->Chinese->Grade 2->Courseware". The searching user can accurately obtain the required information text by clicking on the search directory in the information website. Correspondingly, after a current standard text is captured in the information website, the search directory corresponding to the current standard text can be obtained accordingly. The search directory is generally composed of a search path, which generally contains one or more directory items, and each directory item is equivalent to a classification label.

[0052] Correspondingly, after obtaining the search directory of the current standard text, the directory items of each level in the search directory can be organized according to the preset hierarchical label organization format to obtain the multi-level annotation labels of the current standard text. For example, the first directory item in the search path can be set as a first-level label, and the second directory item in the search path can be set as a second-level label, and so on, to obtain the corresponding multi-level annotation labels. Continuing with the previous example, for the search directory of "Basic Education->Chinese->Second Grade->Courseware", a multi-level annotation label in the form of: "(First-level label: Basic Education)-(Second-level label: Chinese)-(Third-level label: Second Grade)-(Fourth-level label: Courseware)" can be constructed. At the same time, by extracting each directory item included in the search target, the non-hierarchical annotation labels of "Basic Education, Chinese, Second Grade and Courseware" can be obtained.

[0053] S220, input each training sample into the original model, wherein the original model includes a sequence-to-sequence model and a multi-label classifier, and the input of the multi-label classifier is the output of the encoder in the sequence-to-sequence model.

[0054] In this embodiment, after obtaining a plurality of training samples, the plurality of training samples are used to perform model training on the original model of the set structure to obtain a final multi-level label generation model.

[0055] The original model specifically includes: a sequence-to-sequence model and a multi-label classifier. All standard texts in the training samples are input from the input end of the sequence-to-sequence model.

[0056] The sequence-to-sequence model includes: an encoder and a decoder. The encoder is used to analyze the input sequence, obtain the representation vector of the input sequence and output it to the decoder, which generates the output sequence. The input of the multi-label classifier is the output of the encoder in the sequence-to-sequence model.

[0057] A multi-label classifier can be understood as a label classifier implemented based on a multi-task learning (Multi-class) algorithm. The difference is that the types of multi-classification tasks are mutually exclusive. Multi-task learning may have multiple categories (labels) of tasks for each learning sample.

[0058] Typically, the model architecture of the multi-label classifier can be a DNN (Deep Neural Networks), a KNN (K-Nearest Neighbor) model or a decision tree model, etc., which is not limited in this embodiment.

[0059] In this embodiment, the training goal of the sequence-to-sequence model is to output prediction results of multi-layer labels for the input text content, and the training goal of the multi-label classifier is to output prediction results of non-layer labels for the representation vector of the input text.

[0060] S230: According to the difference between the non-hierarchical prediction label of each standard text output by the multi-label classifier and the matching non-hierarchical annotation label, and the difference between the multi-level prediction label of each standard text output by the sequence-to-sequence model and the matching multi-level annotation label, a multi-level label generation model is trained. The output of the multi-level label generation model is the output of the decoder in the trained sequence-to-sequence model.

[0061] It is understandable that the complexity of outputting accurate multi-level labels from a sequence-to-sequence model is much higher than the complexity of outputting accurate non-hierarchical labels from a multi-label classifier. In addition to learning all the labels corresponding to a text, it is also necessary to learn the hierarchical relationship between the above multiple labels, which is difficult to learn and the accuracy cannot be guaranteed. However, the training goal of a multi-label classifier is relatively simple, as long as all the labels corresponding to a text are learned.

[0062] Based on this, the inventor creatively proposed: in the process of training the multi-level label generation model, the input end of the multi-label classifier is connected to the output end of the encoder in the sequence-to-sequence model. At this time, when the multi-label classifier can achieve a good prediction effect, it means that the representation vector output by the encoder in the sequence-to-sequence model can well describe the various implicit text features of the input text. Furthermore, after the representation vector is input to the decoder in the sequence-to-sequence model, the decoder can output a better prediction result of the multi-level label.

[0063] That is, in this embodiment, the difference between the non-hierarchical prediction labels of each standard text output by the multi-label classifier and the matching non-hierarchical annotation labels, as well as the difference between the multi-level prediction labels of each standard text output by the sequence-to-sequence model and the matching multi-level annotation labels are comprehensively considered to construct the loss function of the entire original model, and based on the loss function, the parameters of the original model are adjusted to obtain the final multi-level label generation model.

[0064] The multi-level label generation model is obtained by removing the multi-label classifier from the original model after training.

[0065] The technical solution of the embodiment of the present invention obtains multiple training samples and inputs them into the original model. According to the difference between the non-hierarchical prediction labels of each standard text output by the multi-label classifier in the original model and the matching non-hierarchical annotation labels, as well as the difference between the multi-level prediction labels of each standard text output by the sequence-to-sequence model in the original model and the matching multi-level annotation labels, a multi-level label generation model is trained, and a technical means for performing multi-level label prediction on the target text based on the multi-level label generation model is adopted. On the basis of the structure of the sequence-to-sequence model, the learning of multi-label classification tasks is integrated. On the basis of making full use of the rich semantic information and understanding ability in the sequence-to-sequence model, the potential relationship between the text and its corresponding label is mined to the greatest extent, which can further improve the prediction accuracy of the multi-level label.

[0066] Embodiment 2

[0067] Figure 3 A flowchart of a training method for a multi-level label generation model provided in Embodiment 2 of the present invention. This embodiment is refined based on the above embodiments. In this embodiment, the operation of training the multi-level label generation model is refined according to the difference between the non-hierarchical prediction labels of each standard text output by the multi-label classifier and the matching non-hierarchical annotation labels, as well as the difference between the multi-level prediction labels of each standard text output by the sequence-to-sequence model and the matching multi-level annotation labels.

[0068] Correspondingly, such as Figure 3 As shown, the method includes:

[0069] S310 , obtaining a plurality of training samples, wherein the training samples include standard text, multi-level annotation labels of the standard text, and non-level annotation labels of the standard text.

[0070] S320: Input each training sample into the original model, wherein the original model includes a sequence-to-sequence model and a multi-label classifier, and the input of the multi-label classifier is the output of the encoder in the sequence-to-sequence model.

[0071] Among them, Figure 4 A schematic diagram of the structure of the original model is shown in FIG. Figure 4 As shown, the encoder and the decoder together constitute a sequence-to-sequence model. Further, in this optional implementation, the sequence-to-sequence model is a pre-trained text-to-text model, and the output of the text-to-text model is a label sequence with a set separator as a level identifier.

[0072] Among them, since the pre-trained text-to-text model itself already has very rich semantic information and understanding capabilities, a multi-level label generation model that meets the accuracy requirements can be obtained through simple fine-tuning.

[0073] Furthermore, since the input and output of the text-to-text model are both in text form, it is necessary to convert the discrete label combination into the form of a text sequence so that the hierarchical classification task is aligned with the input and output of the sequence-to-sequence text generation task. Optionally, the data format of the label sequence can be: starting from the root node (top-level label) and ending at the leaf node (bottom-level label), each layer takes the text content of the corresponding label, and the hierarchical identifier used to identify the relationship between the layers is represented by the " / " separator, such as: "People's Livelihood Security / Social Assistance / Temporary Assistance"; or, the candidate items in the entire label tree can be introduced into the label sequence, and the non-compliant label positions are represented by a new separator "_" symbol as a placeholder, such as "People's Livelihood Security / _Social Assistance / ___Temporary Assistance".

[0074] like Figure 4 As shown in the figure, the input of the text-to-text model is a text sequence, which is directly input into the encoder of the text-to-text model, and the encoder encodes the input text sequence into a fixed-length representation vector to capture the semantic information of the text sequence. Then, the decoder of the text-to-text model takes the representation vector as input and gradually generates the output sequence.

[0075] S330 , obtaining a target non-hierarchical prediction label output by the multi-label classifier for the currently input target standard text.

[0076] In this embodiment, a target training sample included in the training samples is taken as an example to describe the model training process for the target training sample.

[0077] The target training samples include: target standard text, non-hierarchical annotation labels of the target standard text, and multi-level annotation labels of the target standard text.

[0078] First, the target standard text is input into the encoder of the text-to-text model, and the representation vector for the target standard text output by the encoder is further input into the multi-label classifier. The multi-label classifier is used to generate a target non-hierarchical prediction label according to the representation vector of the target standard text and output it.

[0079] Among them, the multi-label classifier can realize the multi-label classification task through multiple fully connected layers. The output dimension of the multi-label classifier is the total number of labels, that is, Figure 1 The total number of labels at each level included in the standard label space of the multi-level labels shown. In other words, the total number of label dimensions in the target non-hierarchical prediction label outputs data, each of which represents the score value of the set label, which is generally reflected by the hit probability value.

[0080] S340. Calculate a first loss function according to the target non-hierarchical prediction label and the non-hierarchical annotation label of the target standard text.

[0081] In an optional implementation of this embodiment, a zero-bounded multi-label cross entropy loss (ZMLCE) suitable for multi-label classification can be used as the first loss function L ZMLCE .

[0082] That is:

[0083]

[0084] Among them, N p is a label set consisting of non-hierarchical annotation labels of the target standard text, N n is a label set consisting of labels other than labels without hierarchical annotation in the standard label space of multi-level labels, s yi The score value for each label output by the multi-label classifier.

[0085] S350 , obtaining a target multi-level prediction label output by the sequence-to-sequence model for the currently input target standard text.

[0086] S360: Calculate a second loss function based on the target multi-level prediction labels and the multi-level annotation labels of the target standard text.

[0087] In this embodiment, the cross entropy function crossentropy(.) can be used as the second loss function L Seq2Seq .

[0088] That is:

[0089] L Seq2Seq =crossentropy(Truth,Pred)

[0090] Among them, Truth is the multi-level annotation label of the target standard text, and Pred is the target multi-level prediction label.

[0091] S370. Calculate the target loss function of the original model based on the first loss function and the second loss function, and adjust various model parameters in the original model based on the target loss function to train a multi-level label generation model.

[0092] Among them, the output of the multi-level label generation model is the output of the decoder in the trained sequence-to-sequence model.

[0093] Specifically, the objective function L final It can be the weighted sum of the first loss function and the second loss function, that is:

[0094] L final =L Seq2Seq +αL ZMLCE

[0095] Here, α can be preset according to actual conditions or experimental experience, and this embodiment does not limit this.

[0096] Furthermore, by using multiple training samples in the above manner to adjust each model parameter in the original model respectively, a multi-level label generation model that meets actual application conditions can be obtained when the model training end conditions are met.

[0097] The technical solution of the embodiment of the present invention obtains the target non-hierarchical prediction label output by the multi-label classifier for the currently input target standard text; calculates the first loss function according to the target non-hierarchical prediction label and the non-hierarchical annotation label of the target standard text; obtains the target multi-level prediction label output by the sequence-to-sequence model for the currently input target standard text; calculates the second loss function according to the target multi-level prediction label and the multi-level annotation label of the target standard text; calculates the target loss function of the original model according to the first loss function and the second loss function, and adjusts the various model parameters in the original model according to the target loss function. The technical means can assist in adding multi-label learning tasks in the training process of the multi-level label generation model, use multi-label learning tasks that are easier to train, effectively fine-tune the encoder in the sequence-to-sequence model, and maximize the potential relationship between the text and the label to which it belongs, which can further improve the prediction accuracy of the multi-level labels.

[0098] On the basis of the above embodiments, during the training of the multi-level label generation model, a low-rank adaptive algorithm may be used to perform dimensionality reduction processing on the parameter matrix of each model parameter.

[0099] Low-Rank Adaptation (LoRA) is used to freeze the pre-trained model weights and inject the trainable rank-decomposed matrix into each layer of the machine learning model architecture, greatly reducing the number of trainable parameters for downstream tasks. Specifically, it decomposes the original matrix into the product of two matrices, one of which has a lower rank than the other. At this time, only low-rank matrices need to be used for calculations, which can reduce the number of model parameters, improve training throughput, and perform well in model quality without increasing inference latency.

[0100] Through the above settings, it is possible to fine-tune larger pre-trained models with less video memory consumption, which is in line with the development law of large models.

[0101] Embodiment 3

[0102] Figure 5 This is a flowchart of a text classification method provided in the third embodiment of the present invention. This embodiment is applicable to the case of multi-level label classification of set text. The method can be executed by a text classification device, which can be implemented in the form of hardware and / or software. The text classification device can be configured in an electronic device with data processing function, such as a terminal or a server. Figure 5 As shown, the method includes:

[0103] S510: Obtain target text to be classified.

[0104] The target text is the text to be classified. The classification here can be understood as classifying the target text into a text set with the same multi-level label as the target text according to the multi-level label corresponding to the target text.

[0105] Accordingly, in this embodiment, one or more text sets are included, each text set includes a matching multi-level tag, and all texts in the text set correspond to the same multi-level tag.

[0106] Specifically, for different text classification scenarios, the target text is obtained in different ways. For the recommendation scenario of related text, the target text can be the text that the user clicks to view in real time. After the target text is classified into a text set, other texts with the same multi-level labels can be obtained from the text set for real-time user recommendation.

[0107] S520: Input the target text into a pre-trained multi-level label generation model to obtain a multi-level predicted label of the target text.

[0108] The multi-level label generation model is trained by the training method of the multi-level label generation model as described in any one of the embodiments of the present invention.

[0109] In this embodiment, by inputting the target text into the aforementioned multi-level label generation model, a multi-level predicted label matching the target text can be obtained. The data format of the multi-level predicted label is the same as the data format of the multi-level annotation label in the training sample used to train the multi-level label generation model, typically a label sequence with a set separator as a level identifier.

[0110] S530: Classify the target text into a text set that matches the multi-level prediction labels.

[0111] As described above, after determining the multi-level prediction label corresponding to the target text, the multi-level prediction label can be used as the multi-level label of the target text, and a text set matching the multi-level label can be obtained from multiple pre-established text sets.

[0112] After determining the text set matching the multi-level label, the target text can be classified into the text set matching the multi-level predicted label. Furthermore, according to the actual application scenario, other texts can be obtained from the text set for related recommendations, or the target text can be directly added to the text set, etc., which is not limited in this embodiment.

[0113] The technical solution of the embodiment of the present invention obtains the target text to be classified, inputs the target text into the multi-level label generation model pre-trained by the aforementioned method, and obtains the multi-level prediction label of the target text; the technical means of classifying the target text into a text set that matches the multi-level prediction label can effectively improve the prediction accuracy of the multi-level label of the set text, and then can more accurately classify the text, so as to meet the accurate text classification needs in various scenarios where text classification is required to the greatest extent.

[0114] Embodiment 4

[0115] Figure 6 It is a flowchart of a text classification method provided in Example 4 of the present invention. This embodiment is optimized based on the above embodiments. In this embodiment, the operation of "inputting the target text into a pre-trained multi-level label generation model to obtain multi-level prediction labels of the target text" is further refined.

[0116] Correspondingly, such as Figure 6 As shown, the method specifically includes:

[0117] S610: Obtain target text to be classified.

[0118] S620: Input each text segmentation in the target text into a multi-level label generation model in sequence.

[0119] In this embodiment, the model structure of the multi-level label generation model is a text-to-text model. Therefore, the target text needs to be firstly segmented, and then each segmentation is sequentially input into the multi-level label generation model.

[0120] S630, using the encoder in the multi-level label generation model, sequentially generate encoding features that match each text segmentation, and sequentially input each encoding feature into the decoder in the multi-level label generation model.

[0121] Among them, the above encoding features are the representation vectors of each text segmentation.

[0122] S640. Through the decoder in the multi-level label generation model, whenever the current level label is obtained by decoding the input coding features, all the alternative next level labels limited by the current level label are determined according to the current level label and the preset label tree structure.

[0123] In this embodiment, the inventors further consider that although the prediction accuracy of the multi-level label generation model has been improved, it still produces prediction label results that do not conform to the hierarchical structure. Figure 1The three-level label space shown in the figure predicts a label result that does not conform to the hierarchical structure, such as: "(first-level label: science, education, culture and tourism)-(second-level label: science and technology life)-(third-level label: unemployment insurance)". At the same time, since the entire prediction process follows the prediction method from high-level labels to low-level labels, it is easy to make accurate predictions for relatively sparse low-level labels.

[0124] That is to say, even if a result such as social security is predicted in the secondary label, the probability of getting an incorrect third-level label will be higher because it is necessary to choose among the four labels of "scientific and technological services", "other scientific and technological information", "unemployment insurance" and "industrial and commercial insurance" in the third-level label.

[0125] Based on this, the embodiment of the present invention creatively uses a constrained decoding strategy. After the model training is completed, restrictions are added to the decoding logic processing of the decoder to ensure that the generated labels are arranged in a hierarchical relationship, for example: Figure 1 As shown, after the label "People's Livelihood Guarantee" is generated, the second-level label can only be "Social Security", and the third-level label can only be selected from its sub-labels "Unemployment Insurance" and "Work Injury Insurance", and so on.

[0126] The specific implementation method is: according to the preset tag tree structure (that is, Figure 1 In the label space shown in the figure, a DFA (Deterministic Finite Automata) is constructed. In the model reasoning stage, the beam search strategy of the multi-level label generation model is modified, and the generation probability of the vocabulary entries that the DFA cannot access (that is, all labels except the sub-label set of the last generated label) is set to negative infinity. Adding such a processing process to the reasoning part of the model can strictly ensure that the generated label structure conforms to the structure of the hierarchical label tree, fundamentally avoiding the results of disordered generation order and erroneous content outside the generated label set.

[0127] S650. Through the decoder in the multi-level label generation model, in the vocabulary entry table, the generation probabilities of all labels except the candidate next-level labels are set to negative infinity, and then the vocabulary entry table is used to decode the next-level labels to obtain the multi-level predicted labels of the target text.

[0128] S660: Classify the target text into a text set that matches the multi-level prediction labels.

[0129] The technical solution of the embodiment of the present invention is as follows: each text segmentation in the target text is input into a multi-level label generation model in sequence; an encoder in the multi-level label generation model generates encoding features matching each text segmentation in sequence, and each encoding feature is input into a decoder in the multi-level label generation model in sequence; the decoder in the multi-level label generation model determines all candidate next-level labels limited by the current-level label according to the current-level label and a preset label tree structure whenever the current-level label is decoded according to the input encoding features; the decoder in the multi-level label generation model sets the generation probabilities of all labels except each candidate next-level label in the vocabulary entry table to negative infinity, and then uses the vocabulary entry table to decode the next-level label, which can maximize the guarantee that the multi-level label generation model outputs a predicted label result that conforms to the hierarchical structure, and correspondingly enhances the classification accuracy of the model for low-level labels.

[0130] Embodiment 5

[0131] Figure 7 This is a schematic diagram of the structure of a training device for a multi-level label generation model provided in Embodiment 5 of the present invention. Figure 7 As shown, the device includes: a training sample acquisition module 710, a training sample input module 720 and a model training module 730, wherein:

[0132] The training sample acquisition module 710 is used to acquire multiple training samples, where the training samples include standard text, multi-level annotation labels of the standard text, and non-level annotation labels of the standard text;

[0133] A training sample input module 720 is used to input each training sample into the original model, wherein the original model includes a sequence-to-sequence model and a multi-label classifier, and the input of the multi-label classifier is the output of the encoder in the sequence-to-sequence model;

[0134] A model training module 730 is used to train a multi-level label generation model according to the difference between the non-hierarchical predicted label of each standard text output by the multi-label classifier and the matched non-hierarchical annotated label, and the difference between the multi-level predicted label of each standard text output by the sequence-to-sequence model and the matched multi-level annotated label;

[0135] Among them, the output of the multi-level label generation model is the output of the decoder in the trained sequence-to-sequence model.

[0136] The technical solution of the embodiment of the present invention obtains multiple training samples and inputs them into the original model. According to the difference between the non-hierarchical prediction labels of each standard text output by the multi-label classifier in the original model and the matching non-hierarchical annotation labels, as well as the difference between the multi-level prediction labels of each standard text output by the sequence-to-sequence model in the original model and the matching multi-level annotation labels, a multi-level label generation model is trained, and a technical means for performing multi-level label prediction on the target text based on the multi-level label generation model is adopted. On the basis of the structure of the sequence-to-sequence model, the learning of multi-label classification tasks is integrated. On the basis of making full use of the rich semantic information and understanding ability in the sequence-to-sequence model, the potential relationship between the text and its corresponding label is mined to the greatest extent, which can further improve the prediction accuracy of the multi-level label.

[0137] Based on the above embodiments, the training sample acquisition module 710 can be used to:

[0138] Capture information text from the information website as the current standard text, and obtain the search directory where the current standard text is located;

[0139] According to the preset hierarchical label organization format, the directory items at each level in the search directory are organized to obtain multi-level annotation labels of the current standard text;

[0140] All catalog items included in the search catalog will be determined as non-hierarchical annotation tags for the current standard text.

[0141] Based on the above embodiments, the training sample input module 720 can be used to:

[0142] Get the target non-hierarchical prediction label output by the multi-label classifier for the current input target standard text;

[0143] Calculate a first loss function according to the target non-hierarchical prediction label and the non-hierarchical annotation label of the target standard text;

[0144] Get the target multi-level prediction label output by the sequence-to-sequence model for the current input target standard text;

[0145] Calculate a second loss function according to the target multi-level predicted labels and the multi-level annotation labels of the target standard text;

[0146] According to the first loss function and the second loss function, the target loss function of the original model is calculated, and according to the target loss function, various model parameters in the original model are adjusted.

[0147] On the basis of the above embodiments, the sequence-to-sequence model is a pre-trained text-to-text model, and the output of the text-to-text model is a label sequence with a set separator as a hierarchical identifier;

[0148] And / or, the device further comprises: a dimensionality reduction processing module, configured to:

[0149] During the training of the multi-level label generation model, a low-rank adaptive algorithm is used to reduce the dimension of the parameter matrix of each model parameter.

[0150] The training device for the multi-level label generation model provided in the embodiment of the present invention can execute the training method for the multi-level label generation model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0151] Embodiment 6

[0152] Figure 8 This is a schematic diagram of the structure of a training device for a multi-level label generation model provided in Embodiment 6 of the present invention. Figure 8 As shown, the device includes: a target text acquisition module 810, a multi-level prediction label acquisition module 820 and a target text classification module 830, wherein:

[0153] A target text acquisition module 810 is used to acquire a target text to be classified;

[0154] A multi-level prediction label acquisition module 820 is used to input the target text into a pre-trained multi-level label generation model to obtain the multi-level prediction labels of the target text; wherein the multi-level label generation model is trained by the training method of the multi-level label generation model described in any embodiment of the present invention;

[0155] The target text classification module 830 is used to classify the target text into a text set that matches the multi-level prediction label.

[0156] The technical solution of the embodiment of the present invention obtains the target text to be classified, inputs the target text into the multi-level label generation model pre-trained by the aforementioned method, and obtains the multi-level prediction label of the target text; the technical means of classifying the target text into a text set that matches the multi-level prediction label effectively improves the prediction accuracy of the multi-level label of the set text, and thus can more accurately classify the text, so as to meet the accurate text classification needs in various scenarios where text classification is required to the greatest extent.

[0157] Based on the above embodiments, the multi-level prediction label acquisition module 820 can be specifically used to include:

[0158] Inputting each text segmentation in the target text into a multi-level label generation model in sequence;

[0159] Generate encoding features matching each text segmentation in sequence through the encoder in the multi-level label generation model, and input each encoding feature into the decoder in the multi-level label generation model in sequence;

[0160] Through the decoder in the multi-level label generation model, whenever the current level label is obtained by decoding the input encoding features, all candidate next level labels limited by the current level label are determined according to the current level label and the preset label tree structure;

[0161] Through the decoder in the multi-level label generation model, in the vocabulary entry table, the generation probabilities of all labels except the candidate next-level labels are set to negative infinity, and then the vocabulary entry table is used to decode the next-level labels.

[0162] The text classification device provided in the embodiment of the present invention can execute the text classification method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0163] Embodiment 7

[0164] Fig. 9 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0165] like Fig. 9As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0166] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0167] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a training method for a multi-level label generation model or a text classification method described in an embodiment of the present invention.

[0168] The training method of the multi-level label generation model includes:

[0169] Acquire multiple training samples, the training samples include standard text, multi-level annotation labels of the standard text, and non-level annotation labels of the standard text;

[0170] Input each training sample into the original model, where the original model includes a sequence-to-sequence model and a multi-label classifier, and the input of the multi-label classifier is the output of the encoder in the sequence-to-sequence model;

[0171] According to the difference between the non-hierarchical prediction label of each standard text output by the multi-label classifier and the matching non-hierarchical annotation label, and the difference between the multi-level prediction label of each standard text output by the sequence-to-sequence model and the matching multi-level annotation label, a multi-level label generation model is trained;

[0172] Among them, the output of the multi-level label generation model is the output of the decoder in the trained sequence-to-sequence model.

[0173] The text classification method includes:

[0174] Get the target text to be classified;

[0175] Inputting the target text into a pre-trained multi-level label generation model to obtain a multi-level predicted label of the target text; wherein the multi-level label generation model is trained by the training method of the multi-level label generation model described in any embodiment of the present invention;

[0176] The target text is classified into a text set matching the multi-level predicted labels.

[0177] In some embodiments, a training method for a multi-level label generation model or a text classification method described in an embodiment of the present invention may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the training method for a multi-level label generation model or a text classification method described above as described in an embodiment of the present invention may be executed. Alternatively, in other embodiments, the processor 11 may be configured to execute a training method for a multi-level label generation model or a text classification method as described in an embodiment of the present invention in any other appropriate manner (for example, by means of firmware).

[0178] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0179] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0180] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0181] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0182] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0183] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0184] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0185] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A training method for a multi-level label generation model, characterized in that: include: Acquire multiple training samples, the training samples include standard text, multi-level annotation labels of the standard text, and non-level annotation labels of the standard text; Input each training sample into the original model, where the original model includes a sequence-to-sequence model and a multi-label classifier, and the input of the multi-label classifier is the output of the encoder in the sequence-to-sequence model; According to the difference between the non-hierarchical prediction label of each standard text output by the multi-label classifier and the matching non-hierarchical annotation label, and the difference between the multi-level prediction label of each standard text output by the sequence-to-sequence model and the matching multi-level annotation label, a multi-level label generation model is trained; Among them, the output of the multi-level label generation model is the output of the decoder in the trained sequence-to-sequence model.

2. The method according to claim 1, characterized in that Get multiple training samples, including: Capture information text from the information website as the current standard text, and obtain the search directory where the current standard text is located; According to the preset hierarchical label organization format, the directory items at each level in the search directory are organized to obtain multi-level annotation labels of the current standard text; All catalog items included in the search catalog will be determined as non-hierarchical annotation tags for the current standard text.

3. The method according to claim 1, characterized in that According to the difference between the non-hierarchical prediction label of each standard text output by the multi-label classifier and the matching non-hierarchical annotation label, and the difference between the multi-level prediction label of each standard text output by the sequence-to-sequence model and the matching multi-level annotation label, a multi-level label generation model is trained, including: Get the target non-hierarchical prediction label output by the multi-label classifier for the current input target standard text; Calculate a first loss function according to the target non-hierarchical prediction label and the non-hierarchical annotation label of the target standard text; Get the target multi-level prediction label output by the sequence-to-sequence model for the current input target standard text; Calculate a second loss function according to the target multi-level predicted labels and the multi-level annotation labels of the target standard text; According to the first loss function and the second loss function, the target loss function of the original model is calculated, and according to the target loss function, various model parameters in the original model are adjusted.

4. The method according to any one of claims 1 to 3, characterized in that: The sequence-to-sequence model is a pre-trained text-to-text model, and the output of the text-to-text model is a label sequence with a set separator as a level identifier; and / or During the training of the multi-level label generation model, a low-rank adaptive algorithm is used to reduce the dimension of the parameter matrix of each model parameter.

5. A text classification method, characterized in that: include: Get the target text to be classified; Inputting the target text into a pre-trained multi-level label generation model to obtain a multi-level predicted label of the target text; wherein the multi-level label generation model is trained by the method according to any one of claims 1 to 4; The target text is classified into a text set matching the multi-level predicted labels.

6. The method according to claim 5, characterized in that Input the target text into a pre-trained multi-level label generation model to obtain a multi-level predicted label of the target text, including: Input each text segmentation in the target text into the multi-level label generation model in sequence; Generate encoding features matching each text segmentation in sequence through the encoder in the multi-level label generation model, and input each encoding feature into the decoder in the multi-level label generation model in sequence; Through the decoder in the multi-level label generation model, whenever the current level label is obtained by decoding the input encoding features, all candidate next level labels limited by the current level label are determined according to the current level label and the preset label tree structure; Through the decoder in the multi-level label generation model, in the vocabulary entry table, the generation probabilities of all labels except the candidate next-level labels are set to negative infinity, and then the vocabulary entry table is used to decode the next-level labels.

7. A training device for a multi-level label generation model, characterized in that: include: A training sample acquisition module is used to acquire multiple training samples, the training samples include standard text, multi-level annotation labels of the standard text and non-level annotation labels of the standard text; A training sample input module, used to input each training sample into the original model, wherein the original model includes a sequence-to-sequence model and a multi-label classifier, and the input of the multi-label classifier is the output of the encoder in the sequence-to-sequence model; A model training module, used for training a multi-level label generation model according to the difference between the non-hierarchical prediction label of each standard text output by the multi-label classifier and the matching non-hierarchical annotation label, and the difference between the multi-level prediction label of each standard text output by the sequence-to-sequence model and the matching multi-level annotation label; Among them, the output of the multi-level label generation model is the output of the decoder in the trained sequence-to-sequence model.

8. A text classification device, characterized in that: include: A target text acquisition module is used to acquire the target text to be classified; A multi-level prediction label acquisition module, used to input the target text into a pre-trained multi-level label generation model to obtain the multi-level prediction labels of the target text; wherein the multi-level label generation model is trained by the method according to any one of claims 1 to 4; The target text classification module is used to classify the target text into a text set that matches the multi-level prediction label.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the training method of the multi-level label generation model as described in any one of claims 1-4, or execute the text classification method as described in any one of claims 5-6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a training method for a multi-level label generation model as described in any one of claims 1-4, or to implement a text classification method as described in any one of claims 5-6 when executed.

Images