A classification method for extremely long Chinese texts based on a large language model

Abstract

The present invention discloses a classification method for Chinese super-long texts based on a large language model, comprising the following steps: importing text content through an input device; screening the text content through a preset large language model to obtain pre-classified text; extracting at least two keywords from the pre-classified text content, and importing them into a text classification information table in historical data, determining the proportion of preset words in the keywords in the text classification information table, and replacing the keywords with preset words in the text classification information table according to the proportion; the present invention extracts keywords based on the large language model, and determines the classification information of the text according to the classification information of the keywords in the historical text classification information table. The classification method does not require an overall interpretation of the text, but only requires analysis and processing of the extracted keywords, which can greatly reduce the amount of computer computation, thereby improving the speed and efficiency of text classification.

Description

[0001] Technical preset word group

[0002] The present invention relates to preset character groups for text classification, and in particular to a classification method for Chinese ultra-long texts based on a large language model. Background Art

[0003] Text classification is one of the most important research directions in natural language processing. It is widely used in many scenarios, such as public opinion analysis, spam and email filtering, search engine ranking of web page relevance, information retrieval, data mining, etc.

[0004] In the existing technology, algorithms are usually used to interpret texts, and then the texts are classified according to the overall content of the texts. However, with the advancement of technology, the length of texts is also increasing. When algorithms are continued to be used to read the texts as a whole, the text classification speed is slow and the classification efficiency is low due to the huge text content and the reading speed is limited by the computer's computing speed. Therefore, a classification method for Chinese ultra-long texts based on a large language model is proposed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: how to solve the problem that the text content is huge and the reading speed is limited by the computer's computing speed, resulting in slow text classification and low classification efficiency. A classification method for Chinese ultra-long texts based on a large language model is provided.

[0006] The present invention solves the above technical problems through the following technical solutions, which include the following steps:

[0007] S1. Import text content through input device;

[0008] S2. Filter the text content using a preset large language model to obtain pre-classified text;

[0009] S3, extracting at least two keywords from the pre-classified text content and importing them into the text classification information table in the historical data, determining the proportion of the preset words in the text classification information table in the keywords, and replacing the keywords with the preset words in the text classification information table according to the proportion;

[0010] S4, classifying the text according to the category of the preset vocabulary in the text classification information table, thereby obtaining text classification information;

[0011] S5. Export the text classification information and send the text classification information to a preset terminal.

[0012] Preferably, the S2 specifically includes:

[0013] S21. Read the text content and identify the text title. The specific identification process is as follows:

[0014] First, obtain the text font information, segment the text content according to the text font information, and segment the text of the same type and arranged continuously together;

[0015] Then sort different types of text according to font size;

[0016] Classify the largest font as the main title, the smallest font as the body text, the adjacent titles with the largest font as the summary title, and the remaining fonts into multiple levels of titles in order of font size and classify them separately;

[0017] S22. Mark different text titles with different types;

[0018] S23. Screen and extract text content according to the type tag to produce pre-classified text.

[0019] Preferably, the pre-classified text includes the main title classification text, the keyword includes the first keyword, and S3 specifically includes:

[0020] S31. Obtain the main title classification text and perform denoising on the pre-classified text. The specific denoising process is as follows:

[0021] Import language model data to identify text content in pre-classified text;

[0022] Remove stop words from text content;

[0023] Extracting multiple first keywords from the text content;

[0024] S32: Acquire multiple first keywords in the main title classification text, first classify the multiple first keywords, and then count the multiple first keywords;

[0025] S33, importing the text classification information table in the historical data, and searching the text classification information table for the first keyword one by one;

[0026] S34. Counting the number of first keywords in different categories in the text classification information table based on the search results;

[0027] S35, calculating the share Q of the preset words in the text classification information table among the multiple first keywords. The specific calculation process is:

[0028] Q=A1 / A2

[0029] Wherein, A1 is the number of common words in the multiple first keywords of the preset words in the text classification information table, and A2 is the total number of the multiple first keywords;

[0030] When Q ≥ the preset threshold B, the number of keywords in the same category of the text classification information table is large, and the text content is highly similar to the category, and the first keyword is replaced with the preset vocabulary of the corresponding category;

[0031] When Q is less than the preset threshold B, the number of keywords in the same category of the text classification information table is small, the similarity between the text content and the category is small, and the number of keywords is too small to classify the text content.

[0032] Preferably, the denoising process further includes:

[0033] First, the continuously repeated text content is identified, then the continuously repeated smallest units are screened out, and finally the continuously repeated smallest units are merged to simplify the content of the classified text.

[0034] Preferably, the pre-classified text further includes a summary classified text, and the keyword further includes a second keyword. When Q is less than a preset threshold value B, S3 further includes the following steps:

[0035] S301, obtaining a summary classification text, and extracting a second keyword from the summary classification text;

[0036] S302, screening and counting the number of second keywords;

[0037] S303: Calculate the word frequency P1 of the second keyword. The specific calculation process is as follows:

[0038] P1=C1 / C2

[0039] Where C1 is the number of one of the second keywords, and C2 is the number of the second keywords;

[0040] S304, classifying the second keyword according to the word frequency P1 and determining the priority of the second keyword, when the word frequency P1 is above a preset numerical range, it is a high-level pre-selected word, when P1 is within the preset numerical range, it is a medium-level pre-selected word, and when the word frequency P1 is below the preset numerical range, it is a low-level pre-selected word; and setting the priority as high-level pre-selected word > medium-level pre-selected word > low-level pre-selected word;

[0041] S305 , sequentially including the second keywords into the first keywords according to the priority, and calculating the share Q of the preset words in the text classification information table in the plurality of first keywords, so as to improve the accuracy of text classification.

[0042] Preferably, the pre-classified text further includes the main text classification text, the keyword further includes a third keyword, and S3 further includes the following steps:

[0043] Obtain the main text classification text and extract the third keyword in the main text classification text;

[0044] Filter and count the number of third keywords;

[0045] Calculate the word frequency P2 of the third keyword respectively and classify the third keyword;

[0046] When the word frequency P2 of the third keyword ≥ the preset threshold D, the third keyword appears frequently in the text and is highly representative, and is classified as a high-frequency keyword;

[0047] When the word frequency P2 of the third keyword is less than the preset threshold D, the third keyword has a low frequency of appearance in the text and is weakly representative, and is classified as a low-frequency keyword;

[0048] Assign a correction value E1 to high-frequency keywords and a correction value E2 to low-frequency keywords, stipulate that E1+E2=1, E1>E2, and calculate the participation ability F of the third keyword. The specific calculation process is:

[0049] F=P2 高 *E1+P2 低 *E2

[0050] Among them, P2 高 is the frequency of high-frequency keywords, P2 低 is the frequency of low-level keywords;

[0051] When the participation capacity F ≥ the preset threshold H, the third keyword appears many times in the text and has a great influence on the text classification. It is included in the first keyword and the proportion Q of the preset words in the text classification information table among the multiple first keywords is calculated to improve the accuracy of text classification.

[0052] When the participation ability F is less than the preset threshold H, the third keyword appears less frequently in the text and has little impact on text classification, so it is discarded.

[0053] Preferably, the step S3 further includes the following steps:

[0054] A preset word group is searched for in the text content. When the preset word group content is retrieved, multiple characters before the preset word group are extracted as specific tags. The specific tags are compared with the preset word group in the text classification information table. If the specific tags are different from the preset word group in the text classification information table, the classified text is marked as abnormal.

[0055] Preferably, the S4 specifically includes:

[0056] First, the preset vocabulary is searched in the text classification information table, and then the classification information of the preset vocabulary is searched. Finally, the text to be classified is classified into the classification information of the preset vocabulary, and finally the text classification information is generated according to the classification information of the preset vocabulary.

[0057] Compared with the existing technology, the present invention has the following advantages: the classification method for Chinese ultra-long texts based on a large language model uses a large language model as a basis to screen the text content, thereby extracting keywords, and determining the classification information of the text based on the classification information of the keywords in the historical text classification information table. The classification method does not require an overall interpretation of the text, but only requires analysis and processing of the extracted keywords, which can greatly reduce the computer's computing power, thereby improving the speed and efficiency of text classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0059] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. However, the protection scope of the present invention is not limited to the following embodiment.

[0060] like Figure 1 As shown, this embodiment provides a technical solution: a classification method for Chinese ultra-long text based on a large language model, comprising the following steps:

[0061] S1. Import text content through input device;

[0062] S2. Filter the text content using a preset large language model to obtain pre-classified text;

[0063] S3, extracting at least two keywords from the pre-classified text content and importing them into the text classification information table in the historical data, determining the proportion of the preset words in the text classification information table in the keywords, and replacing the keywords with the preset words in the text classification information table according to the proportion;

[0064] It should be noted that the keyword extraction process is: first convert the words in the text into word vector representation, then use clustering methods such as k-means to cluster the word vectors, calculate the distance between each word and the cluster center, and select the keywords of each cluster as the keywords of the text; commonly used word vector representation methods include fasttext, word2vec, bert, etc.

[0065] It should be noted that the text classification information table is compiled from the classification information of historical texts and is existing data. The text classification information table at least includes fields, major classification directories, minor classification directories, and preset words corresponding to each classification directory for keyword comparison;

[0066] The preset vocabulary is a combination of multiple predicted words reserved for comparison with keywords, and the preset vocabulary includes multiple preset words.

[0067] S4, classifying the text according to the category of the preset vocabulary in the text classification information table, thereby obtaining text classification information;

[0068] S5. Export the text classification information and send the text classification information to a preset terminal.

[0069] This classification method, based on a large language model, screens text content to extract keywords, and determines the classification information of the text based on the classification information of the keywords in the historical text classification information table. This classification method does not require a comprehensive interpretation of the text, but only requires analysis and processing of the extracted keywords, which can greatly reduce the computer's computing power and thus improve the speed and efficiency of text classification.

[0070] Among them, S2 specifically includes:

[0071] S21. Read the text content and identify the text title. The specific identification process is as follows:

[0072] First, obtain the text font information, segment the text content according to the text font information, and segment the text of the same type and arranged continuously together;

[0073] Then sort different types of text according to font size;

[0074] Classify the largest font as the main title, the smallest font as the body text, the adjacent titles with the largest font as the summary title, and the remaining fonts into multiple levels of titles in order of font size and classify them separately;

[0075] S22. Mark different text titles with different types;

[0076] S23. Screen and extract text content according to the type tag to produce pre-classified text.

[0077] Segment the text content and determine the corresponding part of the text through the title, which facilitates the extraction of keywords from the text and improves the efficiency of keyword extraction under the corresponding title.

[0078] The pre-classified text includes the main title classification text, the keyword includes the first keyword, and S3 specifically includes:

[0079] S31. Obtain the main title classification text and perform denoising on the pre-classified text. The specific denoising process is as follows:

[0080] Import language model data to identify text content in pre-classified text;

[0081] Remove stop words from text content;

[0082] Extracting multiple first keywords from the text content;

[0083] It should be noted that stop words refer to certain characters or words that are filtered out before or after processing natural language data or text in information retrieval in order to save storage space and improve search efficiency.

[0084] S32: Acquire multiple first keywords in the main title classification text, first classify the multiple first keywords, and then count the multiple first keywords;

[0085] S33, importing the text classification information table in the historical data, and searching the text classification information table for the first keyword one by one;

[0086] S34. Counting the number of first keywords in different categories in the text classification information table based on the search results;

[0087] S35, calculating the share Q of the preset words in the text classification information table among the multiple first keywords. The specific calculation process is:

[0088] Q=A1 / A2

[0089] Wherein, A1 is the number of common words in the multiple first keywords of the preset words in the text classification information table, and A2 is the total number of the multiple first keywords;

[0090] When Q ≥ the preset threshold B, the number of keywords in the same category of the text classification information table is large, and the text content is highly similar to the category, and the first keyword is replaced with the preset vocabulary of the corresponding category;

[0091] When Q is less than the preset threshold B, the number of keywords in the same category of the text classification information table is small, the similarity between the text content and the category is small, and the number of keywords is too small to classify the text content.

[0092] The text is preliminarily classified by the keywords in the main title. Since the content of the pre-classified text is relatively small, the preliminary classification of the text can be completed at a faster speed, which has the advantage of high speed.

[0093] Furthermore, the denoising process also includes:

[0094] First, the continuously repeated text content is identified, then the continuously repeated smallest units are screened out, and finally the continuously repeated smallest units are merged to simplify the content of the classified text.

[0095] Remove repeated phrases in text content to improve the effectiveness of keyword selection.

[0096] Furthermore, the pre-classified text further includes summary classified text, and the keyword further includes a second keyword. When Q is less than a preset threshold B, S3 further includes the following steps:

[0097] S301, obtaining a summary classification text, and extracting a second keyword from the summary classification text;

[0098] S302, screening and counting the number of second keywords;

[0099] S303: Calculate the word frequency P1 of the second keyword. The specific calculation process is as follows:

[0100] P1=C1 / C2

[0101] Where C1 is the number of one of the second keywords, and C2 is the number of the second keywords;

[0102] S304, classifying the second keyword according to the word frequency P1 and determining the priority of the second keyword, when the word frequency P1 is above a preset numerical range, it is a high-level pre-selected word, when P1 is within the preset numerical range, it is a medium-level pre-selected word, and when the word frequency P1 is below the preset numerical range, it is a low-level pre-selected word; and setting the priority as high-level pre-selected word > medium-level pre-selected word > low-level pre-selected word;

[0103] S305 , sequentially including the second keywords into the first keywords according to the priority, and calculating the share Q of the preset words in the text classification information table in the plurality of first keywords, so as to improve the accuracy of text classification.

[0104] The abstract is a distillation and summary of the main content of the text and is highly representative of the text content. When there are fewer keywords in the main title, the initial classification accuracy of the text is poor. By extracting keywords from the abstract to increase the number of first keywords and classifying the text again, the accuracy of text classification can be improved.

[0105] Furthermore, the pre-classified text also includes the main text classification text, and the keyword also includes the third keyword. S3 also includes the following steps:

[0106] Obtain the main text classification text and extract the third keyword in the main text classification text;

[0107] Filter and count the number of third keywords;

[0108] Calculate the word frequency P2 of the third keyword respectively and classify the third keyword;

[0109] When the word frequency P2 of the third keyword ≥ the preset threshold D, the third keyword appears frequently in the text and is highly representative, and is classified as a high-frequency keyword;

[0110] When the word frequency P2 of the third keyword is less than the preset threshold D, the third keyword has a low frequency of appearance in the text and is weakly representative, and is classified as a low-frequency keyword;

[0111] Assign a correction value E1 to high-frequency keywords and a correction value E2 to low-frequency keywords, stipulate that E1+E2=1, E1>E2, and calculate the participation ability F of the third keyword. The specific calculation process is:

[0112] F=P2 高 *E1+P2 低 *E2

[0113] Among them, P2 高 is the frequency of high-frequency keywords, P2 低 is the frequency of low-level keywords;

[0114] It should be noted that the correction values ​​E1 and E2 are used to restrict the proportion of high-frequency keywords and low-frequency keywords. On this basis, when the frequency proportion of high-frequency keywords and the frequency proportion of low-frequency keywords reach a certain value, the third keyword has a high representativeness in the text, which is enough to affect the classification accuracy of the text.

[0115] When the participation capacity F ≥ the preset threshold H, the third keyword appears many times in the text and has a great influence on the text classification. It is included in the first keyword and the proportion Q of the preset words in the text classification information table among the multiple first keywords is calculated to improve the accuracy of text classification.

[0116] When the participation ability F is less than the preset threshold H, the third keyword appears less frequently in the text and has little impact on text classification, so it is discarded.

[0117] In order to further improve the accuracy of text classification, a representative third keyword is further screened out from the text, and the third keyword is added to the first keyword. When calculating the keyword occupancy Q, the number of keywords in the preset vocabulary can be increased, thereby improving the accuracy of text classification.

[0118] Furthermore, S3 also includes the following steps:

[0119] A preset word group is searched for in the text content. When the preset word group content is retrieved, multiple characters before the preset word group are extracted as specific tags. The specific tags are compared with the preset word group in the text classification information table. If the specific tags are different from the preset word group in the text classification information table, the classified text is marked as abnormal.

[0120] It should be noted that the preset phrases can be highly representative words in a field, etc. The specific labels made from the preset phrases in the text are compared with the preset word groups in the text classification information table to further verify the accuracy of the text classification. When the specific label is different from the preset word groups in the text classification information table, the classified text is marked as abnormal, reminding the user to review or use it with caution. The abnormal classification can be a color mark, such as marking red as an abnormal classification mark.

[0121] Among them, S4 specifically includes:

[0122] First, the preset vocabulary is searched in the text classification information table, and then the classification information of the preset vocabulary is searched. Finally, the text to be classified is classified into the classification information of the preset vocabulary, and finally the text classification information is generated according to the classification information of the preset vocabulary.

[0123] It should be noted that a plurality means at least two.

[0124] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0125] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, technicians of this preset word group can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0126] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are exemplary and are not to be construed as limiting the present invention. A person skilled in the art of the present invention may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A classification method for Chinese ultra-long texts based on a large language model, characterized in that: The following steps are involved: S1. Import text content through input device; S2. Filter the text content using a preset large language model to obtain pre-classified text; S3, extracting at least two keywords from the pre-classified text content and importing them into the text classification information table in the historical data, determining the proportion of the preset words in the text classification information table in the keywords, and replacing the keywords with the preset words in the text classification information table according to the proportion; S4, classifying the text according to the category of the preset vocabulary in the text classification information table, thereby obtaining text classification information; S5. Export the text classification information and send the text classification information to a preset terminal; The pre-classified text includes the main title classification text, the keyword includes the first keyword, and S3 specifically includes: S31, obtaining the main title classification text and performing denoising on the pre-classified text; S32: Acquire multiple first keywords in the main title classification text, first classify the multiple first keywords, and then count the multiple first keywords; S33, importing the text classification information table in the historical data, and searching the text classification information table for the first keyword one by one; S34. Counting the number of first keywords in different categories in the text classification information table based on the search results; S35, calculating the share Q of the preset words in the text classification information table among the multiple first keywords. The specific calculation process is: Q=A1 / A2 Wherein, A1 is the number of common words in the multiple first keywords of the preset words in the text classification information table, and A2 is the total number of the multiple first keywords; When Q ≥ the preset threshold B, the number of keywords in the same category of the text classification information table is large, and the text content is highly similar to the category, and the first keyword is replaced with the preset vocabulary of the corresponding category; When Q is less than the preset threshold value B, the number of keywords in the same category of the text classification information table is small, the similarity between the text content and the category is small, and the number of keywords is too small to classify the text content; The pre-classified text also includes a summary classified text, and the keyword also includes a second keyword. When Q is less than a preset threshold B, S3 further includes the following steps: S301, obtaining a summary classification text, and extracting a second keyword from the summary classification text; S302, screening and counting the number of second keywords; S303: Calculate the word frequency P1 of the second keyword. The specific calculation process is as follows: P1=C1 / C2 Where C1 is the number of one of the second keywords, and C2 is the total number of second keywords; S304, classifying the second keyword according to the word frequency P1 and determining the priority of the second keyword, when the word frequency P1 is above a preset numerical range, it is a high-level pre-selected word, when P1 is within the preset numerical range, it is a medium-level pre-selected word, and when the word frequency P1 is below the preset numerical range, it is a low-level pre-selected word; and setting the priority as high-level pre-selected word > medium-level pre-selected word > low-level pre-selected word; S305 , sequentially including the second keywords into the first keywords according to the priority, and calculating the share Q of the preset words in the text classification information table in the plurality of first keywords, so as to improve the accuracy of text classification.

2. The method for classifying very long Chinese texts based on a large language model according to claim 1, characterized in that: The S2 specifically includes: S21. Read the text content and identify the text title. The specific identification process is as follows: First, obtain the text font information, segment the text content according to the text font information, and segment the text of the same type and arranged continuously together; Then sort different types of text according to font size; Classify the largest font as the main title, the smallest font as the body text, the adjacent titles with the largest font as the summary title, and the remaining fonts into multiple levels of titles in order of font size and classify them separately; S22. Mark different text titles with different types; S23. Screen and extract text content according to the type tag to produce pre-classified text.

3. The method for classifying very long Chinese texts based on a large language model according to claim 1, characterized in that: The specific denoising process in S31 is: Import language model data to identify text content in pre-classified text; Remove stop words from text content; Extract multiple first keywords from the text content.

4. The method for classifying very long Chinese texts based on a large language model according to claim 3, characterized in that: The denoising process further includes: First, the continuously repeated text content is identified, then the continuously repeated smallest units are screened out, and finally the continuously repeated smallest units are merged to simplify the content of the classified text.

5. The method for classifying very long Chinese texts based on a large language model according to claim 4, characterized in that: The pre-classified text also includes the main text classification text, and the keyword also includes a third keyword. S3 also includes the following steps: Obtain the main text classification text and extract the third keyword in the main text classification text; Filter and count the number of third keywords; Calculate the word frequency P2 of the third keyword respectively and classify the third keyword; When the word frequency P2 of the third keyword ≥ the preset threshold D, the third keyword appears frequently in the text and is highly representative, and is classified as a high-frequency keyword; When the word frequency P2 of the third keyword is less than the preset threshold D, the third keyword has a low frequency of appearance in the text and is weakly representative, and is classified as a low-frequency keyword; Assign a correction value E1 to high-frequency keywords and a correction value E2 to low-frequency keywords, stipulate that E1+E2=1, E1>E2, and calculate the participation ability F of the third keyword. The specific calculation process is: <h2 style=";text-align:left;direction:ltr">F=P2<h2 style=";text-align:left;direction:ltr"> 高 <h2 style=";text-align:left;direction:ltr"> *E1+P2<h2 style=";text-align:left;direction:ltr"> 低 <h2 style=";text-align:left;direction:ltr"> *E2 Among them, P2 高 is the frequency of high-frequency keywords, P2 低 is the frequency of low-level keywords; When the participation capacity F ≥ the preset threshold H, the third keyword appears many times in the text and has a great influence on the text classification. It is included in the first keyword and the proportion Q of the preset words in the text classification information table among the multiple first keywords is calculated to improve the accuracy of text classification. When the participation ability F is less than the preset threshold H, the third keyword appears less frequently in the text and has little impact on text classification, so it is discarded.

6. The method for classifying very long Chinese texts based on a large language model according to claim 5, characterized in that: The S3 further comprises the following steps: A preset word group is searched for in the text content. When the preset word group content is retrieved, multiple characters before the preset word group are extracted as specific tags. The specific tags are compared with the preset word group in the text classification information table. If the specific tags are different from the preset word group in the text classification information table, the classified text is marked as abnormal.

7. The method for classifying very long Chinese texts based on a large language model according to claim 6, characterized in that: The S4 specifically includes: First, the preset vocabulary is searched in the text classification information table, and then the classification information of the preset vocabulary is searched. Finally, the text to be classified is classified into the classification information of the preset vocabulary, and finally the text classification information is generated according to the classification information of the preset vocabulary.

Images