Long document multi-dimensional text classification method and system based on scene learning
By constructing query instruction templates and manual annotation data sets, using encoder and document summary to classify long documents multi-dimensional text, the problem of input length limitation in the long document multi-dimensional text classification task is solved, and efficient and flexible text classification effect is achieved.
Patent Information
- Application Number
- CN202411664581.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-20
- Publication Date
- 2025-02-25
AI Technical Summary
In the long document multi-dimensional text classification task, due to the maximum input length of the language model, the effect of scenario learning is limited, and in the multi-dimensional scenario, the difficulty of solving the task is increased and the effective length of the language model is reduced.
By constructing query instruction templates, manually annotating training, development, and testing data sets, the encoder fine-tunes training in the custom data set, using document summary for classification and judgment, and constructing relevant training sample sampling strategies to realize long document multi-dimensional text classification.
The model prediction effect matching the pre-training&fine-tuning paradigm is realized, which reduces the cost of manual labeling, improves the computing efficiency, and has high flexibility and independence.
Smart Images
Figure CN119513319A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of text classification, and specifically, relates to a long document multidimensional text classification method and system based on contextual learning, in particular, a long document multidimensional text classification method based on natural language processing of contextual learning. Background Art
[0002] The text classification task refers to assigning predefined category labels to input text sequences. The text classification task processing process can be roughly decomposed into two steps. First, the feature information of the text sequence is extracted based on the neural network model. Second, the text feature information is passed to the classifier, and the classifier outputs the final category label. In the history of the development of text classification technology, BERT and its variants, such as the Roberta model, are an important node. BERT uses the pre-training & fine-tuning paradigm to handle text classification tasks. In the pre-training stage, BERT learns massive text data in an unsupervised manner, and then fine-tunes the model with a certain amount of manually annotated data when processing specific downstream text classification tasks. This model paradigm has led to significant developments in text classification tasks, but it relies on manually constructing a large amount of training data. In industrial production environments, the condition of high-quality training data is often difficult to meet.
[0003] As a further development of random text classification technology, GPT3 proposed a text generation method to uniformly handle various NLP tasks. Since GPT3 has a huge number of parameters and has been pre-trained with massive data, large-scale language models such as GPT3 store a vast amount of knowledge and have certain semantic understanding and logical reasoning capabilities. Simply input the prompt command to large-scale language models such as GPT3, and the model can generate text to output the expected results of the user.
[0004] Scenario learning refers to inserting several example pairs related to user questions during the prompt instruction construction process.<input,output> These examples provide reference information for the language model to answer user query questions, helping the language model to output reasonable answers to questions in the expected output format.
[0005] Although the research results of situational learning show that the text classification task solution based on situational learning can already match the model prediction effect of the fine-tuning paradigm. However, in the long document classification task, the effect of situational learning is greatly limited due to the limitation of the longest input of the language model, such as the maximum input length of GPT3 is 4096. And in multi-dimensional scenarios, for example, for a long technical document, it is necessary to classify the technical text from multiple dimensions such as manufacturer, technology, industry, etc., which not only increases the difficulty of solving the task, but also further reduces the effective length of the longest input of the language model. This poses a challenge to how to handle long document multi-dimensional text classification tasks based on situational learning.
[0006] The method for hierarchical classification of long text network information proposed in the patent document "Method, system and server for hierarchical classification of long text network information" (CN118535728A) is to first perform data cleaning preprocessing on the long text, which has relatively low computational efficiency and no optimized annotation. Summary of the invention
[0007] In view of the defects in the prior art, the purpose of the present invention is to provide a long document multi-dimensional text classification method and system based on scenario learning.
[0008] A long document multi-dimensional text classification method based on scenario learning provided by the present invention includes:
[0009] Template construction steps: construct query instruction template;
[0010] Annotation construction step: construct manually annotated training, development, and test data sets based on the query instruction template;
[0011] Encoding step: train the encoder based on the annotations;
[0012] Classification judgment step: judge and classify based on the language model;
[0013] Sampling strategy construction step: construct relevant training sample sampling strategy based on classification results;
[0014] Reasoning step: Perform long document label category reasoning based on the query instruction template.
[0015] Preferably, the instruction template describes the multi-dimensional text classification task separately, including the task description x desc , example sample demonstration and document summary x abst .
[0016] The task description x descDescribe the requirements, label sets, and meaning of text classification tasks, including the requirements of text classification tasks, label sets of different dimensions, and task description information.
[0017] The annotation construction step includes:
[0018] Step S2.1: Manually labeled training and test data, with 15 to 30 labels per category;
[0019] Step S2.2: The plain text information of the long document is converted into the document summary x of the long document through the language model in the form of question-answer summary generation. abst ;
[0020] Step S2.3: Randomly divide the corpus into training, development, and test data sets in a ratio of 6:2:2.
[0021] The label includes the long document true label set labels and the long document plain text information x doc .
[0022] The long document plain text information is obtained by file format conversion or document text information extraction.
[0023] In step S2.2, the language model is used to convert <labels,x doc >Data pairs are converted to <labels,x abst >Data pairs.
[0024] Preferably, the encoding step comprises:
[0025] Step S3.1: The encoder is fully fine-tuned in the custom dataset, and the latent vector corresponding to the CLS encoding is taken to convert the document summary x abst Mapped to v abst Document summary vector;
[0026] Step S3.2: Encode the input text sequence, query the encoded vector with the full amount of training samples, and obtain the most relevant N training sample examples;
[0027] Step S3.3: Data pair Any two document summaries in i, j∈{1,N} The similarity between labels i The true value is determined;
[0028] Step S3.4: Fine-tune the training script according to the BERT fine-tuning parameters, train the encoder until the model converges, save the model weights, and save the document summary vector and its related metadata into the vector database.
[0029] The scale of the encoder is base scale or large scale, and the model architecture adopts BERT model, ROBERTA model or Encoder model as the benchmark model.
[0030] Preferably, in the classification judgment step, the language model first collects the document summary x abst The local keyword information in the text is analyzed globally, and the category label is predicted, including:
[0031] Step S4.1: The language model extracts the instruction template of keyword information from the document summary and generates the instruction template for text feature analysis;
[0032] The language model is called by an API interface or deployed locally, and the model parameter quantity is selected from 7B to 130B.
[0033] Step S4.2: Submit document summary x abst As part of the input command, extract the document summary x based on the command template abst Contains keyword information and outputs text feature analysis process;
[0034] During the analysis, for each <labels,x abst > sample pair, get < document summary x abst , keyword list keywords, text feature analysis reasoning, true label set labels> sample pairs.
[0035] Step S4.3: Store all data pairs in the vector database in the form of metadata.
[0036] Preferably, in the sampling strategy construction step, the N data pairs with the highest correlation are queried. The corresponding metadata is taken out from the vector database and assembled into the demonstration part in order of relevance from low to high.
[0037] The metadata includes Data information.
[0038] In the reasoning step, x desc , demonstration, x abst Assemble them into the query instruction template in sequence to generate the query instruction, and pass it to the language model to obtain the category label prediction text output by the language model, and clean it to obtain the category label in the predefined label set.
[0039] A long document multi-dimensional text classification system based on scenario learning provided by the present invention includes:
[0040] Template construction module: construct query instruction template;
[0041] Annotation construction module: builds manually annotated training, development, and test data sets based on query instruction templates;
[0042] Encoding module: train the encoder based on the annotations;
[0043] Classification judgment module: performs judgment and classification based on the language model;
[0044] Sampling strategy building module: build relevant training sample sampling strategies based on classification results;
[0045] Reasoning module: performs long document label category reasoning based on query instruction templates.
[0046] Preferably, the instruction template describes the multi-dimensional text classification task separately, including the task description x desc , example sample demonstration and document summary x abst .
[0047] The task description x desc Describe the requirements, label sets, and meaning of text classification tasks, including the requirements of text classification tasks, label sets of different dimensions, and task description information.
[0048] The annotation construction module includes:
[0049] Module M2.1: Manually labeled training and test data, with 15 to 30 labels per category;
[0050] Module M2.2: The plain text information of a long document is converted into a document summary x of the long document through a language model in the form of question-answer summary generation. abst ;
[0051] Module M2.3: Randomly divide the corpus into training, development, and test data sets in a ratio of 6:2:2.
[0052] The label includes the long document true label set labels and the long document plain text information x doc .
[0053] The long document plain text information is obtained by file format conversion or document text information extraction.
[0054] In the module M2.2, the language model is used to convert <labels,x doc >Data pairs are converted to <labels,x abst >Data pairs.
[0055] Preferably, the encoding module includes:
[0056] Module M3.1: The encoder is fully fine-tuned in the custom dataset, and the latent vector corresponding to the CLS encoding is taken to convert the document summary x abst Mapped to v abst Document summary vector;
[0057] Module M3.2: Encode the input text sequence, query the encoded vector with the full amount of training samples, and obtain the most relevant N training sample examples;
[0058] Module M3.3: Data pair Any two document summaries in i, j∈{1,N} The similarity between labels i The true value is determined;
[0059] Module M3.4: Fine-tune the training script according to the BERT fine-tuning parameters, train the encoder until the model converges, save the model weights, and save the document summary vector and its related metadata into the vector database.
[0060] The scale of the encoder is base scale or large scale, and the model architecture adopts BERT model, ROBERTA model or Encoder model as the benchmark model.
[0061] Preferably, the language model in the classification judgment module first collects the document summary x abst The local keyword information in the text is analyzed globally, and the category label is predicted, including:
[0062] Module M4.1: The language model extracts the instruction template of keyword information from the document summary and generates the instruction template for text feature analysis;
[0063] The language model is called by an API interface or deployed locally, and the model parameter quantity is selected from 7B to 130B.
[0064] Module M4.2: Document summary x abst As part of the input command, extract the document summary x based on the command template abst Contains keyword information and outputs text feature analysis process;
[0065] During the analysis, for each <labels,x abst > sample pair, get < document summary x abst , keyword list keywords, text feature analysis reasoning, true label set labels> sample pairs.
[0066] Module M4.3: Store all data pairs in the vector database as metadata.
[0067] Preferably, the sampling strategy building module queries the N data pairs with the highest correlation. The corresponding metadata is taken out from the vector database and assembled into the demonstration part in order of relevance from low to high.
[0068] The metadata includes Data information.
[0069] The module M6 will be x desc , demonstration, x abst Assemble them into the query instruction template in sequence to generate the query instruction, and pass it to the language model to obtain the category label prediction text output by the language model, and clean it to obtain the category label in the predefined label set.
[0070] Compared with the prior art, the present invention has the following beneficial effects:
[0071] 1. The present invention only requires a small amount of labeled samples to achieve a model prediction effect comparable to the pre-training & fine-tuning paradigm. The cost of manual labeling is low and it has stronger practicality.
[0072] 2. The present invention shares document summary information in one forward propagation, while multiple classifiers will calculate the same document summary information multiple times in the forward propagation, so it has more efficient calculation efficiency.
[0073] 3. The present invention has very high flexibility, and each module component in the system has a high degree of independence. For example, the language model, encoder, vector database, and instruction template are relatively independent, and each component can be customized and optimized according to business needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0075] Figure 1 Schematic diagram of the inference process for category label prediction of multi-dimensional long document model.
[0076] Figure 2 Schematic diagram of instruction template for multi-dimensional task description.
[0077] Figure 3 Schematic diagram of an instruction template for automatically generating document summaries.
[0078] Figure 4 Schematic diagram of the keyword extraction instruction template.
[0079] Figure 5 Schematic diagram of document summary feature analysis instruction template. DETAILED DESCRIPTION
[0080] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.
[0081] The present invention involves large-scale language model, in-context learning, document abstraction generation, text embedding and text classification related technologies. By using a small amount of manually annotated corpus and carefully designing instruction templates based on the idea of context learning, a model prediction effect matching the model fine-tuned with a large amount of manually annotated corpus can be achieved.
[0082] According to a long document multi-dimensional text classification method based on scenario learning provided by the present invention, based on the idea of scenario learning technology method, a customized Prompt instruction is constructed for the long document multi-dimensional text classification scenario, and then the input instruction is passed to the language model to complete the long document multi-dimensional text classification task. Figure 1 Take this as an example, the specific steps are as follows:
[0083] Template construction steps: Build a query instruction template.
[0084] In the instruction construction, a separate and detailed description of the multi-dimensional text classification task is given, which provides overall guidance for how the model outputs the response text. The language model outputs category labels of multiple dimensions in one forward propagation calculation, which improves the computational efficiency.
[0085] Design relevant training sample sampling strategies to build demonstrations. The instruction template consists of three parts: "task description, example sample, query document summary", which are respectively desc , demonstration, x abst The task description x descIt should include the description of the text classification task, a set of labels of different dimensions, and other necessary task description information. Demonstration can be seen as a collection of examples, and demonstration is an important part of the instruction template.
[0086] by Figure 2 For example, the task description describes the requirements of the text classification task, as well as the label set and its meaning. To save space, only three dimensions are listed, and three labels are enumerated for each dimension. In an actual industrial production environment, you can set it yourself according to business needs.
[0087] Annotation construction step: Construct manually annotated training, development, and test datasets.
[0088] Step S2.1: Manually labeled training and test corpus data are labeled according to the standard of 15 to 30 labels per category. doc They represent the true label set of long documents and the plain text information of long documents respectively. Note that the label set contains the category label values under each dimension.
[0089] Step S2.2: The plain text information of the long document is obtained by file format conversion or document text information extraction. Since the average length of the plain text of a long document is long, and the maximum input length of a large-scale language model is limited, the plain text information of the long document x is converted into doc , converted into a document summary x of a long document by using a language model to generate a question-answer summary abst .by Figure 3 Take the example of extracting a document summary from the original plain text of a long document. Through the language model, <labels,x doc >Data pairs are converted to <labels,x abst >Data pairs. The subsequent encoder model training and language model inference phase both use document summaries as input.
[0090] Step S2.3: Finally, the corpus set is randomly divided into training, development, and test data sets in a ratio of 6:2:2.
[0091] Compared with the pre-training & fine-tuning paradigm, only a small number of labeled samples are needed for each label category to achieve a model prediction effect comparable to the pre-training & fine-tuning paradigm. Therefore, the cost of manual labeling is low and it has stronger practicality in actual industrial and service production environments.
[0092] Encoding step: train the encoder.
[0093] Step S3.1: The encoder is used to convert the document summary xabst Mapped to v abst Document summary vector. For the query document summary, based on a certain vector distance metric, find the training samples in the training set that are more similar to the query document summary. There are also many options for the encoder model architecture and the scale of the model parameters, such as BERT model, ROBERTA model, base scale or large scale.
[0094] In the sample selection, a more reasonable automatic selection mechanism is adopted. Specifically, instead of using the open source general Embedding model, the BERT or its variant Encoder model is used as the baseline model based on the downstream text classification task, and the full amount of fine-tuning training is carried out in the custom dataset, and then the latent vector corresponding to the CLS encoding is taken as v abst vector.
[0095] Step S3.2: Encode the input text sequence, and then query the encoded vector with the full amount of training samples to obtain the most relevant N training sample examples, avoiding the significant disadvantage of the sample irrelevance in the random sampling strategy.
[0096] Step S3.3: Custom data pairs created in the annotation construction step Any two document summaries in i, j∈{1,N} The similarity between labels i Specifically, in a certain dimension, if two document summaries have the same category label, the similarity is increased by 1. According to this rule, in multiple dimensions, the more category labels of the same value any two document summaries have, the higher the similarity.
[0097] Step S3.4: After preprocessing the dataset, fine-tune the training script according to the BERT fine-tuning parameters, train the encoder until the model converges, save the model weights, and convert all document summaries in the training set to v abst Document summary vector, and save the document summary vector and its related metadata into the vector database.
[0098] Classification judgment step: Automatically extract x based on the language model abst The process of analyzing keywords and text features contained in the document summary under the condition of taking labels.
[0099] Multi-dimensional classification of long texts is a relatively difficult task. Before the language model makes the final category label prediction, letting the language model first collect local keyword information in the document summary, analyze the text features globally, and finally make a category label prediction can improve the model prediction effect.
[0100] Compared with training multiple text classifiers, in the constructed input instructions, the language model is required to predict the category labels under each dimension at one time, that is, the document summary information is shared in one forward propagation; and multiple classifiers will calculate the same document summary information multiple times in the forward propagation, which has more efficient computing efficiency.
[0101] In more preferred embodiments, the classification determination step includes:
[0102] Step S4.1: Figure 4 , Figure 5 For example, the language model extracts the instruction template of keyword information from the document summary, and the language model generates the instruction template of text feature analysis. In terms of system flexibility, the language model, the core component of the system, uses the API interface call method or the local deployment method, and there are multiple options for the scale of model parameters, such as 7B to 130B.
[0103] Step S4.2: In a single example, a document summary of a long document is used as a component of the input instruction, rather than the original plain text information of the long document as a component of the instruction, so as to avoid the input text being too long.
[0104] Based on the instruction template, the keyword information contained in the document summary is first extracted, and then the text feature analysis process is output. In addition to the document summary, two important components, keyword information and global text feature analysis, are added through the language model. Based on this process, for each <labels,x abst > Sample pairs, can be obtained <x abst , keywords, reasoning, labels> sample pairs, the second item keywords and the third item reasoning represent the keyword list and text feature analysis respectively, that is, in a single example, it contains <document summary, keyword list, text feature analysis, real label set>, so as to simulate the process of human beings making text classification judgments.
[0105] Step S4.3: Store all data pairs in the vector database as metadata to improve the accuracy of model prediction.
[0106] Sampling strategy construction step: Construct relevant training sample sampling strategies.
[0107] The demonstration part of the instruction template is essentially composed of several training sample examples. In the inference stage, for a new query document summary, if the demonstration part contains examples related to the query document summary, the accuracy of the model output will be improved. Cosine similarity is used to measure the similarity between document summary vectors.
[0108] Specifically, the new query document summary is first encoded by the encoder in the encoding step and converted into a vector. Then, the N most relevant document summary vectors are queried in the vector database, and the metadata corresponding to the N most relevant document summary vectors are retrieved from the vector database. Each metadata contains Finally, according to the relevance score, the demonstration part is assembled in sequence according to the sorting criteria from low to high.
[0109] Reasoning step: long document label category reasoning stage.
[0110] Extract plain text information from the input long document, use the language model and the document summary generation instruction template designed in step S2 to extract and generate the document summary x from the plain text information abdt .
[0111] Using the encoder trained in the encoding step, the query document summary x abst Mapped to the query document summary vector and generate a demonstration based on the logic in the sampling strategy construction step.
[0112] x desc , demonstration, x abst The query instruction template in step S1 is assembled in sequence to generate a query instruction, and the query instruction is passed to the language model to obtain the category label prediction text output by the language model.
[0113] The category labels in the predefined label set are cleaned from the category label prediction text.
[0114] The application system is very flexible, and each module component within the system is highly independent. For example, the language model, encoder, vector database, and instruction template are relatively independent, and each component can be customized and optimized according to business needs.
[0115] The present invention also provides a long document multidimensional text classification system based on scenario learning. The long document multidimensional text classification system based on scenario learning can be implemented by executing the process steps of the long document multidimensional text classification method based on scenario learning, that is, those skilled in the art can understand the long document multidimensional text classification method based on scenario learning as a preferred implementation of the long document multidimensional text classification system based on scenario learning.
[0116] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.
[0117] A long document multi-dimensional text classification system based on scenario learning provided by the present invention includes:
[0118] Template construction module: construct query instruction template;
[0119] Annotation construction module: builds manually annotated training, development, and test data sets based on query instruction templates;
[0120] Encoding module: train the encoder based on the annotations;
[0121] Classification judgment module: performs judgment and classification based on the language model;
[0122] Sampling strategy building module: build relevant training sample sampling strategies based on classification results;
[0123] Reasoning module: performs long document label category reasoning based on query instruction templates.
[0124] In more preferred embodiments, the instruction template describes the multi-dimensional text classification task separately, including the task description x desc , example sample demonstration and document summary x abst .
[0125] The task description x desc Describe the requirements, label sets, and meaning of text classification tasks, including the requirements of text classification tasks, label sets of different dimensions, and task description information.
[0126] The annotation construction module includes:
[0127] Module M2.1: Manually labeled training and test data, with 15 to 30 labels per category;
[0128] Module M2.2: The plain text information of a long document is converted into a document summary x of the long document through a language model in the form of question-answer summary generation.abst ;
[0129] Module M2.3: Randomly divide the corpus into training, development, and test data sets in a ratio of 6:2:2.
[0130] The labels include the long document true label set labels, the long document plain text information x doc .
[0131] The long document plain text information is obtained by file format conversion or document text information extraction.
[0132] In the module M2.2, the language model is used to convert <labels,x doc >Data pairs are converted to <labels,x abst >Data pairs.
[0133] In more preferred embodiments, the encoding module includes:
[0134] Module M3.1: The encoder is fully fine-tuned in the custom dataset, and the latent vector corresponding to the CLS encoding is taken to convert the document summary x abst Mapped to v abst Document summary vector;
[0135] Module M3.2: Encode the input text sequence, query the encoded vector with the full amount of training samples, and obtain the most relevant N training sample examples;
[0136] Module M3.3: Data pair Any two document summaries in i, j∈{1,N} The similarity between labels i The true value is determined;
[0137] Module M3.4: Fine-tune the training script according to the BERT fine-tuning parameters, train the encoder until the model converges, save the model weights, and save the document summary vector and its related metadata into the vector database.
[0138] The scale of the encoder is base scale or large scale, and the model architecture adopts BERT model, ROBERTA model or Encoder model as the benchmark model.
[0139] In more preferred embodiments, the language model in the classification judgment module first collects the document summary x abst The local keyword information in the text is analyzed globally, and the category label is predicted, including:
[0140] Module M4.1: The language model extracts the instruction template of keyword information from the document summary and generates the instruction template of text feature analysis;
[0141] The language model is called by an API interface or deployed locally, and the model parameter quantity is selected from 7B to 130B.
[0142] Module M4.2: Document summary x abst As part of the input command, extract the document summary x based on the command template abst Contains keyword information and outputs text feature analysis process;
[0143] During the analysis, for each <labels,x abst > sample pair, get < document summary x abst , keyword list keywords, text feature analysis reasoning, true label set labels> sample pairs.
[0144] Module M4.3: Store all data pairs in the vector database as metadata.
[0145] In more preferred embodiments, the sampling strategy building module queries the N data pairs with the highest correlation. The corresponding metadata is taken out from the vector database and assembled into the demonstration part in order of relevance from low to high.
[0146] The metadata includes Data information.
[0147] In the inference module, x dedc , demonstration, x abst Assemble them into the query instruction template in sequence to generate the query instruction, and pass it to the language model to obtain the category label prediction text output by the language model, and clean it to obtain the category label in the predefined label set.
[0148] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.
Claims
1. A long document multi-dimensional text classification method based on scenario learning, characterized in that: include: Template construction steps: construct query instruction template; Annotation construction step: construct manually annotated training, development, and test data sets based on the query instruction template; Encoding step: train the encoder based on the annotations; Classification judgment step: judge and classify based on the language model; Sampling strategy construction step: construct relevant training sample sampling strategy based on classification results; Reasoning step: Perform long document label category reasoning based on the query instruction template.
2. The long document multi-dimensional text classification method based on scenario learning according to claim 1 is characterized in that: The instruction template describes the multi-dimensional text classification task separately, including the task description x desc , example sample demonstration and document summary x abst ; The task description x desc Describe the requirements, label sets, and meaning of text classification tasks, including the requirements of text classification tasks, label sets of different dimensions, and task description information; The annotation construction step includes: Step S2.1: Manually labeled training and test data, with 15 to 30 labels per category; Step S2.2: The plain text information of the long document is converted into the document summary x of the long document through the language model in the form of question-answer summary generation. abst ; Step S2.3: Randomly divide the corpus into training, development, and test data sets in a ratio of 6:2:2; The label includes the long document true label set labels and the long document plain text information x doc ; The long document plain text information is obtained by file format conversion or document text information extraction; In step S2.2, the language model is used to convert <labels,x doc >Data pairs are converted to <labels,x abst >Data pairs.
3. The long document multi-dimensional text classification method based on scenario learning according to claim 1 is characterized in that: The encoding step comprises: Step S3.1: The encoder is fully fine-tuned in the custom dataset, and the latent vector corresponding to the CLS encoding is taken to convert the document summary x abst Mapped to v abst Document summary vector; Step S3.2: Encode the input text sequence, query the encoded vector with the full amount of training samples, and obtain the most relevant N training sample examples; Step S3.3: Data pair In any two document summaries The similarity between labels i The true value is determined; Step S3.4: Fine-tune the training script according to the BERT fine-tuning parameters, train the encoder until the model converges, save the model weights, and save the document summary vector and its related metadata into the vector database; The scale of the encoder is base scale or large scale, and the model architecture adopts BERT model, ROBERTA model or Encoder model as the benchmark model.
4. The long document multi-dimensional text classification method based on scenario learning according to claim 1 is characterized in that: In the classification judgment step, the language model first collects the document summary x abst The local keyword information in the text is analyzed globally, and the category label is predicted, including: Step S4.1: The language model extracts the instruction template of the keyword information from the document summary and generates the instruction template for text feature analysis; The language model is called by an API interface or deployed locally, and the model parameter quantity is selected from 7B to 130B; Step S4.2: Submit document summary x abst As part of the input command, extract the document summary x based on the command template abst Contains keyword information and outputs text feature analysis process; During the analysis, for each <labels,x abst > sample pair, get < document summary x abst ,keyword list keywords,text feature analysis reasoning,real label set labels>sample pair; Step S4.3: Store all data pairs in the vector database in the form of metadata.
5. The long document multi-dimensional text classification method based on scenario learning according to claim 1 is characterized in that: In the sampling strategy construction step, the N data pairs with the highest correlation are queried. And take out the corresponding metadata from the vector database, sort them from low to high in terms of relevance and assemble them into the demonstration part; The metadata includes Data information; In the reasoning step, x desc , demonstration, x abst Assemble them into the query instruction template in sequence to generate the query instruction, and pass it to the language model to obtain the category label prediction text output by the language model, and clean up to obtain the category label in the predefined label set.
6. A long document multi-dimensional text classification system based on contextual learning, characterized in that: include: Template construction module: construct query instruction template; Annotation construction module: builds manually annotated training, development, and test data sets based on query instruction templates; Encoding module: train the encoder based on the annotations; Classification judgment module: performs judgment and classification based on the language model; Sampling strategy building module: build relevant training sample sampling strategies based on classification results; Reasoning module: performs long document label category reasoning based on query instruction templates.
7. The long document multi-dimensional text classification system based on scenario learning according to claim 6 is characterized in that: The instruction template describes the multi-dimensional text classification task separately, including the task description x desc , example sample demonstration and document summary x abst ; The task description x desc Describe the requirements, label sets, and meaning of text classification tasks, including the requirements of text classification tasks, label sets of different dimensions, and task description information; The annotation construction module includes: Module M2.1: Manually labeled training and test data, with 15 to 30 labels per category; Module M2.2: The plain text information of a long document is converted into a document summary x of the long document through a language model in the form of question-answer summary generation. abst ; Module M2.3: Randomly divide the corpus into training, development, and test data sets in a ratio of 6:2:2; The label includes the long document true label set labels and the long document plain text information x doc ; The long document plain text information is obtained by file format conversion or document text information extraction; In the module M2.2, the language model is used to convert <labels,x doc >Data pairs are converted to <labels,x abst >Data pairs.
8. The long document multi-dimensional text classification system based on scenario learning according to claim 6 is characterized in that: The encoding module comprises: Module M3.1: The encoder is fully fine-tuned in the custom dataset, and the latent vector corresponding to the CLS encoding is taken to convert the document summary x abst Mapped to v abst Document summary vector; Module M3.2: Encode the input text sequence, query the encoded vector with the full amount of training samples, and obtain the most relevant N training sample examples; Module M3.3: Data pair In any two document summaries The similarity between labels i The true value is determined; Module M3.4: Fine-tune the training script according to the BERT fine-tuning parameters, train the encoder until the model converges, save the model weights, and save the document summary vector and its related metadata into the vector database; The scale of the encoder is base scale or large scale, and the model architecture adopts BERT model, ROBERTA model or Encoder model as the benchmark model.
9. The long document multi-dimensional text classification system based on scenario learning according to claim 6 is characterized in that: The language model in the classification judgment module first collects the document summary x abst The local keyword information in the text is analyzed globally, and the category label is predicted, including: Module M4.1: The language model extracts the instruction template of keyword information from the document summary and generates the instruction template for text feature analysis; The language model is called by an API interface or deployed locally, and the model parameter quantity is selected from 7B to 130B; Module M4.2: Document summary x abst As part of the input command, extract the document summary x based on the command template abst Contains keyword information and outputs text feature analysis process; During the analysis, for each <labels,x abst > sample pair, get < document summary x abst ,keyword list keywords,text feature analysis reasoning,real label set labels>sample pair; Module M4.3: Store all data pairs in the vector database as metadata.
10. The long document multi-dimensional text classification system based on scenario learning according to claim 6, characterized in that: The sampling strategy construction module queries the N data pairs with the highest correlation And take out the corresponding metadata from the vector database, sort them from low to high in terms of relevance and assemble them into the demonstration part; The metadata includes Data information; In the inference module, x desc , demonstration, x abst Assemble them into the query instruction template in sequence to generate the query instruction, and pass it to the language model to obtain the category label prediction text output by the language model, and clean up to obtain the category label in the predefined label set.
Citation Information
Patent Citations
Method, system and server for hierarchically classifying long text network information
CN118535728A