A Slicing Method for Non-standard Documents

Abstract

The present invention relates to the technical field of text processing, and specifically discloses a slicing processing method for non-standard documents, comprising the following steps: S1: determining a first keyword and a second keyword; S2: obtaining a first word vector and a second word vector, and determining the degree of association between a sub-paragraph and a chapter name based on the first word vector and the second word vector; S3: determining a pending paragraph according to the degree of association, sorting the pending paragraphs to obtain a paragraph sorting; determining the chapter name corresponding to the target paragraph, and determining the pending paragraph again according to the new chapter name; determining that the document is normalized when there is no pending paragraph, and slicing the normalized document. The present invention improves the accuracy of recall by normalizing and slicing non-standard documents.

Description

Technical Field

[0001] The present invention relates to the technical field of text processing, and in particular to a slicing processing method for non-standard documents. Background Art

[0002] Recall is a key concept in information retrieval, which is used to measure the ability to retrieve relevant information from massive data. In practical applications, non-standard documents often bring challenges to the recall process. Non-standard documents refer to documents that do not meet standardization requirements in terms of format, such as missing chapter titles. Such problems cause incoherent information within the document, thus affecting the accurate recall of information by large language models.

[0003] At the same time, large language models have limitations on context windows when processing long texts. In other words, the model can only process a certain number of words or tokens (for example, 2048 tokens). When the document length exceeds this limit, the model may lose context information before and after. This will result in the model being unable to effectively process the part beyond the context window during retrieval, resulting in an inability to accurately locate the target information. This limitation affects the recall accuracy of the model.

[0004] Therefore, how to improve the recall accuracy when retrieving non-standard documents using large language models has become an urgent problem to be solved. Summary of the invention

[0005] The purpose of the present invention is to provide a non-standard document slicing processing method to solve the following technical problems:

[0006] Improve recall accuracy when retrieving non-canonical documents using large language models.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A method for slicing a non-standard document includes the following steps:

[0009] S1: Mark the paragraphs corresponding to a single chapter in the document as sub-paragraphs, extract keywords in the sub-paragraphs based on the TF-IDF algorithm, use them as the first keywords, segment the chapter name, and obtain the second keywords;

[0010] S2: Get the word vector of the first keyword and record it as the first word vector wf=(C1, C2, ..., C n ), C n represents the value of the first word vector on the nth dimension, where n represents the total number of dimensions, and the word vector of the second keyword is obtained and recorded as the second word vector ws=(D1, D2, …, D n ), Dn represents the value of the second word vector in the nth dimension, and determines the degree of association between the sub-paragraph and the chapter name based on the first word vector and the second word vector;

[0011] S3: marking the sub-paragraphs corresponding to the correlation between the sub-paragraphs and the chapter names that is less than a preset correlation threshold as pending paragraphs, and sorting the pending paragraphs in descending order according to the correlation degree to obtain a paragraph sorting;

[0012] Taking the first pending paragraph in the paragraph sorting as the target paragraph, obtaining the text summary of the target paragraph, taking the text summary as the chapter name f, determining the second word vector F corresponding to the chapter name f, determining the degree of association between the pending paragraph and the chapter name f based on the first word vector and the second word vector F, and determining the pending paragraph again;

[0013] Repeat the above steps until there are no pending paragraphs, and the document is normalized. Take a chapter in the normalized document as a document slice, and the document is sliced.

[0014] As a further solution of the present invention: the step S3 further includes the following steps:

[0015] Determine a text summary c of a subparagraph in the document slice, segment the text summary c to obtain a third keyword, and record a word vector of the third keyword as a third word vector;

[0016] The user's search text is obtained, and the search text is segmented to obtain a fourth keyword. The word vector of the fourth keyword is recorded as a fourth word vector, and the degree of association K between the search text and the text summary c is determined based on the third word vector and the fourth word vector. When the degree of association K is greater than or equal to a preset search association degree threshold, the corresponding sub-paragraph is extracted and displayed to the user.

[0017] As a further solution of the present invention: in step S2, the process of determining the degree of association between the subparagraph and the chapter name specifically includes:

[0018] Calculate the first vector , Cn a represents the value of the a-th first word vector in the n-th dimension;

[0019] Calculate the second vector ,Dn b represents the value of the b-th second word vector in the n-th dimension;

[0020] The cosine similarity between the first vector and the second vector is calculated and used as the correlation degree between the subparagraph and the chapter name.

[0021] As a further solution of the present invention: in the step S3, a text summary of the target paragraph is obtained based on the TextRank algorithm, and stop words in the text summary are removed based on a predefined stop word list.

[0022] As a further solution of the present invention: in the step S2, the first word vector is obtained based on a deep learning model, and the deep learning model includes Word2Vec, GloVe and FastText.

[0023] As a further solution of the present invention: in step S3, the pending paragraphs with the same degree of association are sorted according to the order in which the pending paragraphs appear in the document.

[0024] Beneficial effects of the present invention: In this solution, the document is subdivided and marked as sub-paragraphs, so that the subsequent pre-trained model can understand the content of each part more accurately, and the TF-IDF algorithm is used to extract keywords from the sub-paragraphs. The TF-IDF algorithm can identify the core subject words of the sub-paragraphs by measuring the importance of words in the document and the entire corpus. Keyword extraction helps to identify the most important concepts in each sub-paragraph, avoid information redundancy, and improve processing efficiency. The TF-IDF algorithm belongs to the prior art and will not be described in detail here. Then, the chapter name is segmented to obtain the keywords therein, which helps to understand the main content of the chapter; extracting the keywords of the chapter name lays the foundation for the subsequent correlation calculation and helps to correctly classify the sub-paragraphs into corresponding chapters; converting the keywords of the sub-paragraphs into word vector representations, which can convert the semantic information of the text into numerical form, which is convenient for calculation and comparison. Similarly, converting the keywords of the chapter name into word vector representations, based on the first word vector (wf) and the second word vector (ws), by calculating the similarity between them, the correlation between the sub-paragraph and the chapter theme is evaluated, and the semantic matching degree between the sub-paragraph and the chapter name can be judged to avoid Avoid misclassification of sub-paragraphs and improve the document's structured degree and content coherence; then, mark the sub-paragraphs with a correlation degree lower than the preset threshold as pending paragraphs, identify the content that does not match the current chapter or is unclear in its attribution, and sort the pending paragraphs in descending order according to the degree of correlation, so that the paragraphs most likely to be reclassified are processed first. Then, select the pending paragraph with the highest correlation in the sorting, extract its text summary, and use the summary as the new chapter name f. This step is to create a suitable chapter name for those paragraphs that are missing a chapter title. Based on the new chapter name f, obtain its word vector F. Recalculate the degree of association with the first word vectors of other pending paragraphs to evaluate whether they should be included in new chapters. Repeat the above steps through loop iteration until all pending paragraphs are classified to ensure that there are no missed sub-paragraphs in the document and complete the standardization of the document. Through iterative processing, gradually improve the chapter division of the document to make the document structure more reasonable and facilitate subsequent document segmentation. Generate new chapter names using the summaries of the pending paragraphs to solve the problem of missing chapter titles in non-standard documents. The reclassified and named chapters ensure the consistency between the sub-paragraph content and the chapter theme, and enhance the logic of the document.Finally, each chapter is sliced ​​as an independent document, so that the length of each slice fits the context window limit of the large language model, avoiding context truncation and information loss when the model processes ultra-long texts. At the same time, slicing according to chapters ensures that each slice is a complete chapter with coherent content and clear themes, and the integrity of the information will not be destroyed due to segmentation. The slicing technology enables the model to effectively process various parts of long documents and integrate them to achieve understanding of the entire document. The content after slicing maintains context coherence, and the model can obtain complete information within a limited context window, thereby improving the accuracy of recall. The present invention improves the accuracy of recall by normalizing and slicing non-standard documents. ; BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below in conjunction with the accompanying drawings.

[0026] Figure 1 It is a flow chart of a non-standard document slicing processing method of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] See also Figure 1 As shown, the present invention is a slicing processing method for non-standard documents, comprising the following steps:

[0029] S1: Mark the paragraphs corresponding to a single chapter in the document as sub-paragraphs, extract keywords in the sub-paragraphs based on the TF-IDF algorithm, use them as the first keywords, segment the chapter name, and obtain the second keywords;

[0030] S2: Get the word vector of the first keyword and record it as the first word vector wf=(C1, C2, ..., C n ), C n represents the value of the first word vector on the nth dimension, where n represents the total number of dimensions, and the word vector of the second keyword is obtained and recorded as the second word vector ws=(D1, D2, …, D n ), D n represents the value of the second word vector in the nth dimension, and determines the degree of association between the sub-paragraph and the chapter name based on the first word vector and the second word vector;

[0031] S3: marking the sub-paragraphs corresponding to the correlation between the sub-paragraphs and the chapter names that is less than a preset correlation threshold as pending paragraphs, and sorting the pending paragraphs in descending order according to the correlation degree to obtain a paragraph sorting;

[0032] Taking the first pending paragraph in the paragraph sorting as the target paragraph, obtaining the text summary of the target paragraph, taking the text summary as the chapter name f, determining the second word vector F corresponding to the chapter name f, determining the degree of association between the pending paragraph and the chapter name f based on the first word vector and the second word vector F, and determining the pending paragraph again;

[0033] Repeat the above steps until there are no pending paragraphs, and the document is normalized. Take a chapter in the normalized document as a document slice, and the document is sliced.

[0034] It should be noted that the document is subdivided and marked as sub-paragraphs so that the subsequent pre-trained model can understand the content of each part more accurately, and the TF-IDF algorithm is used to extract keywords from the sub-paragraphs. The TF-IDF algorithm can identify the core subject words of the sub-paragraph by measuring the importance of words in the document and the entire corpus. Keyword extraction helps to identify the most important concepts in each sub-paragraph, avoid information redundancy, and improve processing efficiency. The TF-IDF algorithm belongs to the existing technology and will not be described in detail here. Then, the chapter name is segmented and the keywords are obtained, which helps to understand the main content of the chapter; extracting the keywords of the chapter name lays the foundation for the subsequent correlation calculation and helps to correctly classify the sub-paragraphs into the corresponding chapters; converting the keywords of the sub-paragraphs into word vector representations. The word vector can convert the semantic information of the text into numerical form, which is easy to calculate and compare. Similarly, converting the keywords of the chapter name into word vector representations, based on the first word vector (wf) and the second word vector (ws), by calculating the similarity between them, the correlation between the sub-paragraph and the chapter theme can be evaluated. The semantic matching degree between the sub-paragraph and the chapter name can be judged to avoid the sub-paragraph. 's misclassification to improve the document's structured degree and content coherence; then, mark the sub-paragraphs with a correlation degree lower than the preset threshold as pending paragraphs, identify the content that does not match the current chapter or is unclear in its attribution, and sort the pending paragraphs in descending order according to the degree of correlation, so that the paragraphs that are most likely to be reclassified are processed first. Then, select the pending paragraph with the highest correlation in the sorting, extract its text summary, and use the summary as the new chapter name f. This step is to create a suitable chapter name for those paragraphs that are missing a chapter title. Based on the new chapter name f, obtain its word vector F and compare it with the new chapter name f. The first word vector of the pending paragraphs is recalculated to evaluate whether they should be included in the new chapter. The above steps are repeated through loop iteration until all the pending paragraphs are classified, ensuring that there are no missing sub-paragraphs in the document and completing the normalization of the document. Through iterative processing, the chapter division of the document is gradually improved to make the document structure more reasonable and facilitate subsequent document segmentation. The summary of the pending paragraph is used to generate a new chapter name, which solves the problem of missing chapter titles in non-standard documents. The reclassified and named chapters ensure the consistency between the sub-paragraph content and the chapter theme, and enhance the logic of the document.Finally, each chapter is sliced ​​as an independent document, so that the length of each slice fits the context window limit of the large language model, avoiding context truncation and information loss when the model processes ultra-long texts. At the same time, slicing according to chapters ensures that each slice is a complete chapter with coherent content and clear themes, and the integrity of information will not be destroyed due to segmentation. The slicing technology enables the model to effectively process various parts of long documents and integrate them to achieve understanding of the entire document. The content after slicing maintains context coherence, and the model can obtain complete information within a limited context window, improving the accuracy of recall. ;

[0035] In another preferred embodiment of the present invention, the step S3 further includes the following steps:

[0036] Determine a text summary c of a subparagraph in the document slice, segment the text summary c to obtain a third keyword, and record a word vector of the third keyword as a third word vector;

[0037] The user's search text is obtained, and the search text is segmented to obtain a fourth keyword. The word vector of the fourth keyword is recorded as a fourth word vector, and the degree of association K between the search text and the text summary c is determined based on the third word vector and the fourth word vector. When the degree of association K is greater than or equal to a preset search association degree threshold, the corresponding sub-paragraph is extracted and displayed to the user.

[0038] It is worth noting that the text summary c of the sub-paragraph in the document slice is segmented to obtain the third keyword, which is converted into the third word vector. These keywords represent the main content of the sub-paragraph; the search text input by the user is segmented to obtain the fourth keyword, which is converted into the fourth word vector, representing the user's search intention; based on the third word vector and the fourth word vector, the correlation degree K between them is calculated to measure the matching degree between user needs and document content. When the correlation degree K is greater than or equal to the preset search correlation degree threshold, the corresponding sub-paragraph is extracted and displayed to the user; through the calculation of correlation at the semantic level, the user's search intention can be understood more accurately and matched with the document content; through precise correlation calculation and content matching, the accuracy of the search results is ensured, making up for the shortcomings of the large language model in long context processing.

[0039] In another preferred embodiment of the present invention, in step S2, the process of determining the degree of association between the subparagraph and the chapter name specifically includes:

[0040] Calculate the first vector , Cn a represents the value of the a-th first word vector in the n-th dimension;

[0041] Calculate the second vector , Dn b represents the value of the b-th second word vector in the n-th dimension;

[0042] Calculate the cosine similarity between the first vector and the second vector, and use it as the degree of association between the sub-paragraph and the chapter name.

[0043] It can be understood that the cosine similarity calculation method belongs to the prior art and will not be elaborated here.

[0044] It should be noted that all degrees of association in this application are calculated with reference to the above method.

[0045] In another preferred embodiment of the present invention, in step S3, a text summary of the target paragraph is obtained based on the TextRank algorithm, and stop words in the text summary are removed based on a predefined stop word list.

[0046] It should be noted that the TextRank algorithm is a text summary algorithm based on a graph model, which can automatically extract key information in a paragraph and generate a highly summarized text summary. The most representative sentences or keywords are extracted from the target paragraph through the TextRank algorithm to form a concise and informative summary; stop words refer to common words filtered out in text processing, such as "de", "le", "zai", etc., which contribute less to the main semantics of the text. By removing stop words, the text summary is made more refined, retaining the words that contribute the most to the semantics, and improving the accuracy of subsequent word vector calculations.

[0047] In another preferred embodiment of the present invention, in step S2, the first word vector is obtained based on a deep learning model, and the deep learning model includes Word2Vec, GloVe, and FastText.

[0048] It should be noted that these deep learning models can capture the complex semantic and syntactic relationships between words. They are trained on a large-scale corpus and learn the distributed representations of words, mapping words into a high-dimensional continuous vector space, so that words with similar semantics are close to each other in the vector space; secondly, the word vectors generated by deep learning models have high dimensions and rich semantic information, which can reflect complex semantics such as hyponymy and association relationships between words. This makes the calculation of the degree of association more accurate, reduces the risk of misclassifying sub-paragraphs, and improves the quality of document normalization; at the same time, deep learning models can also be fine-tuned for specific domains and languages to generate domain-specific word vectors, enhancing the applicability and universality of the solution, making it applicable not only to Chinese documents but also to document processing in other languages and professional fields; among them, the second word vector, the third word vector, and the fourth word vector are all obtained in the above manner.

[0049] In another preferred embodiment of the present invention, in step S3, the pending paragraphs with the same degree of association are sorted according to the order in which the pending paragraphs appear in the document.

[0050] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A slicing method for non-standard documents, characterized in that: The following steps are involved: S1: Mark the paragraphs corresponding to a single chapter in the document as sub-paragraphs, extract keywords in the sub-paragraphs based on the TF-IDF algorithm, use them as the first keywords, segment the chapter name, and obtain the second keywords; S2: Get the word vector of the first keyword and record it as the first word vector wf=(C1, C2, ..., C n ), C n represents the value of the first word vector on the nth dimension, where n represents the total number of dimensions, and the word vector of the second keyword is obtained and recorded as the second word vector ws=(D1, D2, …, D n ), D n represents the value of the second word vector in the nth dimension, and determines the degree of association between the sub-paragraph and the chapter name based on the first word vector and the second word vector; S3: marking the sub-paragraphs corresponding to the correlation between the sub-paragraphs and the chapter names that is less than a preset correlation threshold as pending paragraphs, and sorting the pending paragraphs in descending order according to the correlation degree to obtain a paragraph sorting; Taking the first pending paragraph in the paragraph sorting as the target paragraph, obtaining the text summary of the target paragraph, taking the text summary as the chapter name f, determining the second word vector F corresponding to the chapter name f, determining the degree of association between the pending paragraph and the chapter name f based on the first word vector and the second word vector F, and determining the pending paragraph again; Repeat the above steps until there are no pending paragraphs, and the document is normalized. Take a chapter in the normalized document as a document slice, and the document is sliced.

2. According to the non-standard document slicing method of claim 1, it is characterized in that: The step S3 further includes the following steps: Determine a text summary c of a subparagraph in the document slice, segment the text summary c to obtain a third keyword, and record a word vector of the third keyword as a third word vector; The user's search text is obtained, and the search text is segmented to obtain a fourth keyword. The word vector of the fourth keyword is recorded as a fourth word vector, and the degree of association K between the search text and the text summary c is determined based on the third word vector and the fourth word vector. When the degree of association K is greater than or equal to a preset search association degree threshold, the corresponding sub-paragraph is extracted and displayed to the user.

3. The method for slicing a non-standard document according to claim 1, characterized in that: In step S2, the process of determining the degree of association between the subparagraph and the chapter name specifically includes: Calculate the first vector , Cn a represents the value of the a-th first word vector in the n-th dimension; Calculate the second vector ,Dn b represents the value of the b-th second word vector in the n-th dimension; The cosine similarity between the first vector and the second vector is calculated and used as the correlation degree between the subparagraph and the chapter name.

4. The method for slicing a non-standard document according to claim 1, characterized in that: In step S3, a text summary of the target paragraph is obtained based on the TextRank algorithm, and stop words in the text summary are removed based on a predefined stop word list.

5. The method for slicing a non-standard document according to claim 1, characterized in that: In step S2, the first word vector is obtained based on a deep learning model, and the deep learning model includes Word2Vec or GloVe or FastText.

6. According to claim 1, a non-standard document slicing method is characterized in that: In step S3, the pending paragraphs with the same degree of association are sorted according to the order in which the pending paragraphs appear in the document.

Images