Document classification method and electronic device
By representing documents hierarchically as paragraphs and sentences, and utilizing a self-attention mechanism and the pre-trained model RoBERTa, the problem of insufficient paragraph and document representation in existing technologies is solved, thereby improving the accuracy and interpretability of document classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CAS OF CHENGDU INFORMATION TECH CO LTD
- Filing Date
- 2022-11-17
- Publication Date
- 2026-07-31
AI Technical Summary
There is currently no universally applicable pre-trained language model to represent coarse-grained data, such as paragraphs and document input features, which leads to insufficient performance of classification models.
The document is divided into multiple independent paragraph units according to the paragraph organization structure, and sentences, paragraphs and documents are represented layer by layer through a self-attention mechanism. Word vector encoding is performed using the pre-trained model RoBERTa to build a hierarchical classification model.
The granularity of document representation is refined, which improves the performance of the classification model. The hierarchical representation significantly improves the accuracy and interpretability of document classification.
Smart Images
Figure CN115688010B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of natural language processing, and more specifically, to a document classification method and an electronic device. Background Technology
[0002] In classification tasks, feature representation of input features is the foundation of all models. As information and data are generated and accumulated, there is an increasing demand for appropriate representation of input features at different granularities. In recent years, large-scale pre-trained models such as BERT (Bidirectional Encoder Representation from Transformers) have shone brightly in the field of NLP (Natural Language Processing), gradually gaining dominance in word vector representation. Their outstanding performance has also driven the vigorous development of large-scale pre-trained language models. However, for coarser-grained data, such as paragraph representation and document representation, there is still no universal pre-trained language model. Summary of the Invention
[0003] The purpose of this disclosure is to provide a document classification method and electronic device to address the technical problem in the prior art that there is currently no universally applicable pre-trained language model to represent the input features of coarse-grained data, such as paragraphs and documents.
[0004] To achieve the above objectives, the first aspect of this disclosure provides a document classification method, including:
[0005] Divide the document into multiple independent paragraph units according to its paragraph organization structure;
[0006] Each paragraph unit is segmented into sentence units while maintaining the relative positions between the sentence units.
[0007] For each independent paragraph unit, each sentence unit is segmented into words to obtain a list of word blocks;
[0008] The word block list is encoded to obtain a high-dimensional word vector for each word;
[0009] The high-dimensional word vectors are sequentially fed into the self-attention module, and the resulting self-attention outputs are averaged to obtain the sentence vectors.
[0010] The sentence vectors are sequentially fed into the self-attention module, and the obtained self-attention outputs are averaged to obtain the paragraph vectors until the paragraph vectors of all paragraphs in the document are obtained.
[0011] The paragraph vectors are sequentially fed into the self-attention module, and the resulting self-attention outputs are averaged to obtain the document vector.
[0012] The document vector is fed into a classifier to obtain the classification probability distribution of the document.
[0013] Optionally, the document can be divided into multiple independent paragraph units according to its paragraph organization structure, including:
[0014] The document is split according to line breaks or blank lines to obtain a list of natural language text containing multiple paragraphs.
[0015] Optionally, each paragraph unit is segmented into sentence units while maintaining the relative positions between the sentence units, including:
[0016] The sentence segmenter is used to split each paragraph into multiple independent sentences according to the sentence-end symbols, resulting in a list of natural language text containing multiple sentences.
[0017] Optionally, each sentence unit is segmented into words to obtain a list of word blocks, including:
[0018] The sentence is segmented using a word segmenter to obtain a list of word blocks containing multiple words.
[0019] Optionally, the word block list is encoded to obtain a high-dimensional word vector for each word, including:
[0020] The word block list is passed through a word segmenter corresponding to a pre-trained language model to obtain the label information of each word block in the dictionary;
[0021] The word vector matrix is obtained by sequentially passing the label information of each word block in the dictionary through the word encoding layer. Where N represents the number of word blocks in the word vector matrix, d embed This represents the dimension of the word vector.
[0022] Optionally, the high-dimensional word vectors are sequentially fed into a self-attention module, and the resulting self-attention outputs are averaged to obtain the sentence vector, including:
[0023] The word vector matrix I is input into the self-attention module, where the calculation process of the self-attention module is as follows:
[0024] in, Given the input word vector matrix, and The learnable parameters are used to map the input word vectors to a high-dimensional space. In this high-dimensional space, the attention score matrix is obtained by querying the vector matrix Q and the key vector matrix K. The self-attention output is obtained by performing matrix multiplication between the score matrix and the value vector matrix.
[0025] After obtaining the self-attention output, the sentence vector matrix is obtained by averaging the self-attention output matrix column by column.
[0026] Optionally, the sentence vectors are sequentially fed into the self-attention module, and the resulting self-attention outputs are averaged to obtain the paragraph vector, including:
[0027] The sentence vector matrix S is input into the self-attention module, where the calculation process of the self-attention module is as follows:
[0028] After obtaining the self-attention output, the paragraph vector is obtained by averaging the self-attention output matrix column by column.
[0029] Optionally, the paragraph vectors are sequentially fed into the self-attention module, and the resulting self-attention outputs are averaged to obtain the document vector, including:
[0030] The paragraph vector matrix G is input into the self-attention module, where the calculation process of the self-attention module is as follows: in, Given the input word vector matrix, and These are learnable parameters;
[0031] After obtaining the self-attention output, the document vector is obtained by averaging the self-attention output matrix column by column.
[0032] Optionally, the document vector is fed into a classifier to obtain the classification probability distribution of the document, including:
[0033] Inputting the document vector representation into the classifier yields the classification probability distribution: The document category is:
[0034] A second aspect of this disclosure provides an electronic device, comprising:
[0035] A memory on which computer programs are stored;
[0036] A processor for executing the computer program in the memory to implement the steps of the method described in the first aspect.
[0037] In this embodiment, document classification is based on a pre-trained model and an attention mechanism. First, the document is divided into multiple independent paragraph units according to certain rules. Each paragraph unit is then segmented into sentence units, maintaining their relative positions. Next, a hierarchical classification model is constructed. This involves segmenting sentence units into words to obtain high-dimensional word vector representations, using an attention mechanism to obtain sentence representations, and then using the sentence representations of each paragraph to obtain paragraph representations. Finally, the paragraph representations and the attention mechanism are used to obtain the document representation. Subsequently, the document vectors are passed through a classifier, and the classification probability distribution of the document is output. This embodiment addresses the technical problem in the prior art where there is currently no universally applicable pre-trained language model to represent coarse-grained data, such as paragraph and document input features. It refines the granularity of the hierarchical document representation model, representing documents at the word, sentence, paragraph, and document levels, significantly improving the performance of the classification model.
[0038] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description
[0039] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:
[0040] Figure 1 This is a block diagram illustrating a document classification method according to an exemplary embodiment;
[0041] Figure 2 This is a schematic diagram of the structure of a layered model according to an exemplary embodiment;
[0042] Figure 3 This is a flowchart illustrating a document classification method according to an exemplary embodiment;
[0043] Figure 4 This is a schematic diagram of the structure of a sentence encoder according to an exemplary embodiment;
[0044] Figure 5 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation
[0045] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.
[0046] like Figure 1 As shown, Figure 1This is a block diagram illustrating a document classification method according to an exemplary embodiment. First, the document data is preprocessed by dividing it layer by layer in a top-down manner, using sentences as units. The smallest unit of data is then processed by a language model to obtain a word vector matrix. Subsequently, the data is fed into the document classification model. After layered calculation, the document classification probability distribution is obtained. The loss is calculated, and the error is backpropagated to update the network parameters. After multiple rounds of training, the model acquires the ability to assign importance weights to elements in different layers. Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of a hierarchical model according to an exemplary embodiment, including word embedding, sentence encoder, paragraph encoder, document encoder, and document vector.
[0047] In this embodiment, by leveraging downstream tasks, representation models for sentences and documents can be obtained simultaneously during model training. Furthermore, using the pre-trained language model RoBERTa as a word vector encoder significantly simplifies the training of the representation model. For document representation, since documents are naturally hierarchically organized—from words to sentences, from sentences to paragraphs, and from paragraphs to articles—a hierarchical model can be used for hierarchical representation. Considering the differences in importance of different paragraph vectors within a document, an attention mechanism is employed to learn the importance weights of different paragraphs. Using these importance weights to perform a weighted average of the paragraph vectors further enhances the rationality and interpretability of the document representation.
[0048] Next, the document classification method in the embodiments of this disclosure will be described.
[0049] like Figure 3 As shown, Figure 3 This is a flowchart illustrating a document classification method according to an exemplary embodiment, the method comprising the following steps.
[0050] Step 301: Divide the document into multiple independent paragraph units according to the document's paragraph organization structure.
[0051] In this embodiment of the disclosure, data processing is layered. For any document data, tokenization is performed layer by layer in a top-down manner. For long documents, paragraphs are generally independent of each other and contain one or more line breaks. Therefore, the document can be split according to the line break '\n' or the blank line '\r\n' to obtain a list of paragraphs.
[0052] Step 302: Divide each paragraph unit into sentence units while keeping the relative positions of the sentence units unchanged.
[0053] For any paragraph, a sent tokenizer is used to separate it into sentences. For example, for Chinese data, the sent tokenizer may divide the paragraph into sentences based on sentence-ending symbols such as "." or "!". After dividing the paragraph into sentences, a list of sentences is obtained.
[0054] Step 303: Segment each sentence unit into words to obtain a list of word blocks.
[0055] Step 304: Encode the word block list to obtain a high-dimensional word vector for each word.
[0056] Words are the smallest unit of data. For each sentence list, a word segmenter is used to segment the sentences, resulting in a list of word chunks containing multiple words. Then, a pre-trained language model is used to convert the words into word vectors, resulting in a word vector matrix. Specifically, the word chunk list can be passed through a word segmenter corresponding to the pre-trained language model to obtain the dictionary label information of each word chunk in the list; then, the dictionary label information of each word chunk is sequentially passed through a word encoding layer to obtain the word vector matrix. Where N represents the number of word blocks in the word vector matrix, d embed This represents the dimension of the word vector.
[0057] Step 305: The high-dimensional word vectors are sequentially fed into the self-attention module, and the obtained self-attention outputs are averaged to obtain the sentence vectors.
[0058] Specifically, the word vector matrix I is input into the self-attention module, where the calculation process of the self-attention module is as follows: in, Given the input word vector matrix, and The learnable parameters are used to map the input word vectors to a high-dimensional space. In this high-dimensional space, the attention score matrix is obtained by querying the vector matrix Q and the key vector matrix K. The self-attention output is obtained by performing matrix multiplication between the score matrix and the value vector matrix.
[0059] After obtaining the self-attention output, the sentence vector is obtained by averaging the matrix column by column.
[0060] In step 305, a component that is general and shares weights across the sentence encoding layer is formed, which can effectively reduce model parameters. The structure of the sentence encoder is as follows: Figure 4 As shown, since the encoder structures of different layers differ only in the input, the other layers are not shown separately.
[0061] Step 306: The sentence vectors are sequentially fed into the self-attention module, and the obtained self-attention outputs are averaged to obtain the paragraph vectors, until the paragraph vectors of all paragraphs in the document are obtained.
[0062] Specifically, the sentence vector matrix S is input into the self-attention module, where the calculation process of the self-attention module is as follows: After obtaining the self-attention output, the paragraph vector is obtained by averaging the matrix column by column.
[0063] Repeat steps 303, 304, 305, and 306 until all paragraph vectors of the document are obtained.
[0064] Step 307: Feed all paragraph vectors into the self-attention module in sequence, and average the self-attention outputs to obtain the document vector.
[0065] Specifically, the paragraph vector matrix G is input into the self-attention module, where the calculation process of the self-attention module is as follows: in, Given the input word vector matrix, and These are learnable parameters; after obtaining the self-attention output, the document vector is obtained by averaging the matrix column by column.
[0066] Step 308: The document vector is fed into a classifier to obtain the classification probability distribution of the document.
[0067] Specifically, the document vector representation is input into the classifier to obtain the classification probability distribution: The document category is:
[0068] Then, based on the document's labels, the error between the classification probability distribution calculated by the model and the correct label (ground truth) is calculated, and the model parameters are updated using backpropagation of the error.
[0069] Figure 5 This is a block diagram illustrating an electronic device 500 according to an exemplary embodiment. For example... Figure 5 As shown, the electronic device 500 may include a processor 501 and a memory 502. The electronic device 500 may also include one or more of a multimedia component 503, an input / output (I / O) interface 504, and a communication component 505.
[0070] The processor 501 controls the overall operation of the electronic device 500 to complete all or part of the steps described above. The memory 502 stores various types of data to support the operation of the electronic device 500; this data may include, for example, instructions for any application or method operating on the electronic device 500, and application-related data. The multimedia component 503 may include a screen and audio components. The I / O interface 504 provides an interface between the processor 501 and other interface modules. The communication component 505 enables wired or wireless communication between the electronic device 500 and other devices.
[0071] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the document classification method described above. For example, the computer-readable storage medium may be the memory 502 including the program instructions described above, which may be executed by the processor 501 of the electronic device 500 to complete the document classification method described above.
[0072] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described document classification method when executed by the programmable device.
[0073] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.
[0074] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.
[0075] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.
Claims
1. A method of classifying a document, characterized by, include: Divide the document into multiple independent paragraph units according to its paragraph organization structure; Each paragraph unit is segmented into sentence units while maintaining the relative positions between the sentence units. For each independent paragraph unit, each sentence unit is segmented into words to obtain a list of word blocks; Encoding the list of word blocks yields a high-dimensional word vector for each word, including: The word block list is passed through a word segmenter corresponding to a pre-trained language model to obtain the label information of each word block in the dictionary; The label information of each word block in the dictionary is sequentially passed through the word coding layer to obtain a word vector matrix wherein N represents the number of word blocks of the word vector matrix, dembed represents the dimension of the word vector. The high-dimensional word vector is sequentially sent to a self-attention module, and the obtained self-attention output is averaged as a sentence vector, and the word vector matrix The input self-attention module, wherein the calculation process of the self-attention module is: ; in, ; Given the input word vector matrix, , and These are learnable parameters used to map the input word vectors to a high-dimensional space, as shown in the formula above. , , These are the query vector matrices. Key vector matrix Sum value vector matrix The projection dimension, in the high-dimensional space, through and To obtain the attention score matrix, and through the score matrix The self-attention output is obtained by performing matrix multiplication between the fraction matrix and the value vector matrix. After obtaining the self-attention output, the sentence vector matrix is obtained by averaging the self-attention output matrix by column ; The sentence vectors are sequentially fed into the self-attention module, and the resulting self-attention outputs are averaged to obtain the paragraph vectors. This process continues until the paragraph vectors of all paragraphs in the document are obtained, including: The sentence vector matrix Input the self-attention module, where the calculation process of the self-attention module is as follows: ; After obtaining the self-attention output, a paragraph vector is obtained by averaging the self-attention output matrix by column ; The paragraph vectors are sequentially fed into the self-attention module, and the resulting self-attention outputs are averaged to obtain the document vector, including: The paragraph vector matrix Input the self-attention module, where the calculation process of the self-attention module is as follows: ;in, , Given the input word vector matrix, , and These are learnable parameters; After obtaining the self-attention output, the document vector is obtained by averaging the self-attention output matrix by column ; The document vector is fed into a classifier to obtain the classification probability distribution of the document.
2. The method of claim 1, wherein, The document is divided into multiple independent paragraph units according to its paragraph organization structure, including: The document is split according to line breaks or blank lines to obtain a list of natural language text containing multiple paragraphs.
3. The method of claim 1, wherein, Each paragraph unit is segmented into sentence units while maintaining the relative positions between the sentence units, including: The sentence segmenter is used to split each paragraph into multiple independent sentences according to the sentence-end symbols, resulting in a list of natural language text containing multiple sentences.
4. The method of claim 1, wherein, Each sentence unit is segmented into words to obtain a list of word blocks, including: The sentence is segmented using a word segmenter to obtain a list of word blocks containing multiple words.
5. The method of claim 1, wherein, The document vector is fed into a classifier to obtain the classification probability distribution of the document, including: Inputting the document vector representation into the classifier yields the classification probability distribution: ,in For the first The original category score corresponding to the document type. For the goal of The original classification scores for each category; the document categories are: .
6. An electronic device, comprising: include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-4.