Document segmentation method for self-adaptive slices generated through large model retrieval enhancement

Abstract

The invention discloses a document segmentation method for self-adaptive slices generated by large model retrieval enhancement, and relates to the technical field of large model retrieval enhancement generation, the method comprises the following steps: obtaining a to-be-segmented document, and segmenting the to-be-segmented document according to a title type to obtain at least one group of original blocks; according to the information density and the theme change degree corresponding to any original block, the optimal segmentation number of the original block is calculated; and performing segmentation processing on the original blocks according to the optimal segmentation number. According to the method, firstly, the document is segmented according to the hierarchical titles, then the information density and the theme change degree under the hierarchical titles are calculated, and the optimal segmentation size under the hierarchical titles is automatically calculated by taking the hierarchical titles as units, so that self-adaptive segmentation of the document is guided, and the effects of subsequent retrieval and task generation are improved.

Description

Background Art

[0002] Retrieval Augmented Generation is a technology applied in the field of knowledge content retrieval. It combines vector databases and large language models, and aims to enhance the generation capabilities of large models through retrieval technology. The construction process of the vector database is: document preparation → document segmentation → slice vectorization → storage. Document segmentation is to optimize the construction of the vector database and improve the generation quality of the large language model. In this technology, the large model will provide reference answers based on the document segmentation content after retrieval sorting, so document segmentation is particularly important for the answer quality of the large model.

[0003] In terms of document segmentation technology in retrieval enhanced generation, previous methods mainly include two methods: segmentation based on a fixed number of words and segmentation based on specified punctuation marks.

[0004] The method based on fixed word count segmentation often cannot adapt well to the changes in content and structure of different documents. Since the paragraph length and content of different documents vary greatly, fixed word count segmentation may lead to inaccurate segmentation results, affecting subsequent document processing and analysis.

[0005] In view of the limitations of traditional document segmentation technology, further optimization and improvement are needed to meet the higher requirements for document segmentation technology. Summary of the invention

[0006] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a document segmentation method for adaptive slicing generated by large model retrieval enhancement, which is as follows:

[0007] 1) In the first aspect, the present invention provides a document segmentation method for adaptive slicing generated by large model retrieval enhancement, and the specific technical scheme is as follows:

[0008] Obtaining a document to be segmented, and segmenting the document to be segmented according to the title type to obtain at least one set of original segments;

[0009] According to the information density and the degree of topic change corresponding to any original block, the optimal number of divisions of the original block is calculated;

[0010] The original block is segmented according to the optimal number of segments.

[0011] The beneficial effects of the document segmentation method of adaptive slicing generated by large model retrieval enhancement provided by the present invention are as follows:

[0012] First, the document is segmented according to the hierarchical titles, and then the information density and topic variation under the hierarchical titles are calculated. Taking the hierarchical titles as units, the optimal segmentation size under the hierarchical titles is automatically calculated to guide the adaptive segmentation of the document, so as to improve the effect of subsequent retrieval and generation tasks.

[0013] Based on the above solution, the present invention can also be improved as follows.

[0014] Furthermore, the information density is calculated as follows:

[0015]

[0016] Among them, T i represents the information density corresponding to the i-th original block, N d is the total number of words after deduplication in the i-th original block, t i N d The weight of the i-th word in the deduplicated text obtained by the TF-IDF algorithm, n i Represents N d The frequency of occurrence of the i-th word in the deduplicated text, N all Represents the number of all words in the i-th original block.

[0017] Furthermore, the topic change degree is calculated as follows:

[0018] D i =D JS (v i-1 ,v i );

[0019] Among them, D i is the topic change degree corresponding to the i-th original block, D JS is the Jensen-Shannon divergence, v i-1 represents the vector distribution corresponding to the i-1th original cut, v i Represents the vector distribution of the current i-th original slice.

[0020] Furthermore, the optimal number of cuts is calculated as follows:

[0021]

[0022] Among them, S i is the optimal number of segments corresponding to the ith original segment, C represents the total number of characters corresponding to the ith original segment, and α, β, and γ are adjustment parameters.

[0023] 2) In the second aspect, the present invention also provides a document segmentation system based on adaptive slicing of hierarchical titles in large model retrieval enhancement generation, and the specific technical solution is as follows:

[0024] The acquisition module is used to: acquire the document to be segmented, and segment the document to be segmented according to the title type to obtain at least one set of original segments;

[0025] The calculation module is used to calculate the optimal number of cuts of any original cut block according to the information density and the theme change degree corresponding to the original cut block;

[0026] The segmentation module is used to segment the original block according to the optimal segmentation quantity.

[0027] Based on the above solution, the present invention can also be improved as follows.

[0028] Furthermore, the information density is calculated as follows:

[0029]

[0030] Among them, T i represents the information density corresponding to the i-th original block, N d is the total number of words after deduplication in the i-th original block, t i N d The weight of the i-th word in the deduplicated text obtained by the TF-IDF algorithm, n i Represents N d The frequency of occurrence of the i-th word in the deduplicated text, N all Represents the number of all words in the i-th original block.

[0031] Furthermore, the topic change degree is calculated as follows:

[0032] D i =D JS (v i-1 ,v i );

[0033] Among them, D i is the topic change degree corresponding to the i-th original block, D JS is the Jensen-Shannon divergence, v i-1 represents the vector distribution corresponding to the i-1th original cut, v i Represents the vector distribution of the current i-th original slice.

[0034] Furthermore, the optimal number of cuts is calculated as follows:

[0035]

[0036] Among them, S iis the optimal number of segments corresponding to the ith original segment, C represents the total number of characters corresponding to the ith original segment, and α, β, and γ are adjustment parameters.

[0037] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, wherein the processor is coupled to a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above methods.

[0038] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor so that a computer implements any of the above methods.

[0039] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0041] Figure 1 A schematic diagram of a flow chart of a document segmentation method for adaptive slicing generated by large model retrieval enhancement according to an embodiment of the present invention;

[0042] Figure 2 The figure is a structural frame diagram of an electronic device. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0044] like Figure 1 As shown, a document segmentation method for adaptive slicing generated by large model retrieval enhancement according to an embodiment of the present invention comprises the following steps:

[0045] S1, obtaining a document to be segmented, and segmenting the document to be segmented according to the title type to obtain at least one set of original segments;

[0046] S2, according to the information density and the subject change degree corresponding to any original block, calculate the optimal number of divisions of the original block;

[0047] S3, dividing the original block according to the optimal number of divisions.

[0048] The beneficial effects of the document segmentation method of adaptive slicing generated by large model retrieval enhancement provided by the present invention are as follows:

[0049] First, the document is segmented according to the hierarchical titles, and then the information density and topic variation under the hierarchical titles are calculated. Taking the hierarchical titles as units, the optimal segmentation size under the hierarchical titles is automatically calculated to guide the adaptive segmentation of the document, so as to improve the effect of subsequent retrieval and generation tasks.

[0050] The title types are:

[0051] Simple title: This title usually uses only one word or phrase to express the subject and is suitable for internal documents or simple reports, such as "Notice", "Request", "Report", etc.

[0052] Complete title: Use a complete sentence or phrase to express the subject, which is suitable for external documents or formal reports, such as "Report on XXX Matter", "Investigation Report on XXX Issue", etc.

[0053] Prefix title: Add a prefix before the simple title or full title to express the subject more accurately, such as "Urgent Notice", "Important Report", "Recommendation Report on XXX Issue", etc.

[0054] Standard title: This title consists of three elements: the issuing authority, the reason and the name of the document type. It is usually used in formal documents, such as "Notice on the launch of XXX activities."

[0055] Binary title: It consists of the issuing authority and the name of the document type, or the reason and the name of the document type. It is suitable for relatively concise official document titles.

[0056] News-style titles: Mainly used in briefings, investigative reports, speeches and other types of documents. They are very casual and the text is lively. Some have only one title, while others have two titles, such as "XXX event ended successfully" and "XXX unit's 2023 annual work summary."

[0057] First-level title: The serial number is "I.", the font is 4-point bold, and it is on a single line without punctuation at the end. The first-level title is usually the largest title in the article, used to summarize the main content or chapters of the article.

[0058] Secondary title: The serial number is "(I)", the font size is the same as the main text, it is on a separate line, and there is no punctuation at the end. The secondary title is usually a sub-argument under the primary title, used to further subdivide the content.

[0059] The optimal number of splits is calculated as follows:

[0060]

[0061] The adjustment parameters α, β and γ are determined as follows:

[0062] Determine the first information density of the document to be segmented, and determine whether the first information density is lower than the preset minimum information density. If so, further determine whether the current original segment also includes the previous original segment. If so, extract the first keyword of the title of the current original segment and the second keyword of the current original segment, and extract the third keyword of the title of the previous original segment and the fourth keyword of the previous original segment, determine the first correlation and the first similarity between the first keyword and the third keyword, determine the second correlation and the second similarity between the second keyword and the fourth keyword, determine a first similarity score based on the first correlation and the first similarity, determine a second similarity score based on the second correlation and the second similarity, determine a first total score of the current original segment based on the first similarity score and the second similarity score, and determine the range of α based on the first total score.

[0063] The process of determining similarity scores is as follows:

[0064] The similarity score is determined by determining the weights corresponding to the correlation and similarity.

[0065] The process of determining the first total score of the current original cut block based on the first similarity score and the second similarity score is as follows:

[0066] Determine the first overall score based on the title weight and the text weight.

[0067] If the first information density is not lower than the preset minimum information density, or the current original block does not include the previous original block, the preset α is used as α in the optimal number of divisions of the current original block, and further judged whether β and γ need to be adjusted.

[0068] The preceding original segment refers to an original segment before the current original segment among a plurality of original segments arranged in the order of the text of the document to be segmented.

[0069] When any original cut contains more than a preset number of professional terms and more than a preset length of text description of technical specific content, the target number of professional terms is determined, and the target length of the text description is determined. Based on the table corresponding to the number of professional terms and the first weight, and the table corresponding to the length of the text description, the first target weight corresponding to the target number of professional terms and the second target weight corresponding to the target length of the text description are determined. According to the first target weight and the second target weight, combined with the number of professional terms and the text description, the second total score corresponding to the original cut is determined. The range of β is determined according to the second total score.

[0070] If any original block does not contain more than the preset number of professional terms, or does not contain a text description of technical specific content exceeding the preset length, the preset β is used as the β in the optimal number of divisions of the current original block, and further determines whether γ needs to be adjusted.

[0071] According to the third keyword and the fourth keyword, it is determined whether the current original block and the previous original block are completely different topics, that is, it is determined whether the overlap between the third keyword and the fourth keyword is lower than the preset overlap. If it is lower, γ is adjusted. If it is not lower than the preset overlap, the preset γ is used as the γ in the optimal number of segments of the current original block.

[0072] Furthermore, the information density is calculated as follows:

[0073]

[0074] Among them, T i represents the information density corresponding to the i-th original block, N d is the total number of words after deduplication in the i-th original block, t i N d The weight of the i-th word in the deduplicated text obtained by the TF-IDF algorithm, n i Represents N d The frequency of occurrence of the i-th word in the deduplicated text, N all Represents the number of all words in the i-th original block.

[0075] Furthermore, the topic change degree is calculated as follows:

[0076] D i =D JS (v i-1 ,v i );

[0077] Among them, D i is the topic change degree corresponding to the i-th original block, D JS is the Jensen-Shannon divergence, v i-1 represents the vector distribution corresponding to the i-1th original cut, v i Represents the vector distribution of the current i-th original slice.

[0078] Furthermore, the optimal number of cuts is calculated as follows:

[0079]

[0080] Among them, S iis the optimal number of segments corresponding to the ith original segment, C represents the total number of characters corresponding to the ith original segment, and α, β, and γ are adjustment parameters.

[0081] 1. Document preparation: Collect the documents that need to be processed.

[0082] Preprocessing: Split each document into hierarchical titles (also known as sub-titles, interjections, and sub-titles).

[0083] Adaptive Slicing:

[0084] Calculation formula:

[0085]

[0086] in:

[0087] S i : The optimal slice size of the i-th level title;

[0088] C: constant, representing the total number of characters in the text under the i-th level title;

[0089] α: Adjustment parameter used to adjust the basic slice size. When there is no obvious difference in information density in the document, this parameter determines the default slice size;

[0090] If the document has low information density as a whole and the topic does not vary much, then α will play a dominant role and provide a reasonable default slice size. For example, for a narrative prose, since its information density is relatively uniform, α determines the slice size of most paragraphs.

[0091] β: Adjustment parameter used to weight information density T i If β is larger, it means that the information density has a greater impact on the slice size;

[0092] For example, in a technical document, some paragraphs may contain a lot of terminology and technical details (high information density). In this case, a larger β will make the slices of these paragraphs smaller to better capture important information.

[0093] γ: Adjustment parameter used to weight the topic change score D i If γ is larger, it means that the change of the subject has a greater impact on the slice size;

[0094] If the document suddenly turns from discussing one technology to a completely different topic, increasing γ will cause the paragraphs near these turning points to be cut into smaller slices to maintain the thematic coherence of each slice.

[0095] T i: Information density of the title at the i-th level. Calculated based on TF-IDF (term frequency-inverse document frequency). Paragraphs with higher information density usually contain more key information, so smaller slices may be needed to ensure that the information is not overly dispersed;

[0096] D i : The topic change degree of the i-th level title. It is calculated by Jensen-Shannon divergence, which measures the degree of topic transition from one level title to the next. A high topic change score means that the paragraph may be a transition between two different topics and should be cut more finely to maintain thematic coherence.

[0097] Specifically, T i Represents the information density of the title text at the i-th level, ranging from 0 to 1, and the formula is:

[0098]

[0099] in:

[0100] N d Represents the total number of all words in the input text after deduplication;

[0101] t i Represents this N d The weight of the i-th word among the deduplicated words obtained by the TF-IDF algorithm;

[0102] n i Represents the frequency of occurrence of the word;

[0103] N all Represents the number of all words in the paragraph;

[0104] D i Represents the theme change degree of the i-th level title, ranging from 0 to 1, and the formula is:

[0105] D i =D JS (v i-1 ,v i )

[0106] in:

[0107] D JS is the Jensen-Shannon divergence, which measures the difference between two probability distributions;

[0108] v i-1 Represents the vector distribution of the previous (i-1) text;

[0109] v i Represents the vector distribution of the current i-th paragraph of text;

[0110] The vector is generated by the general BERT model;

[0111] Note that when i is the first segment, D i =0.

[0112] The present invention can segment each paragraph, and on the basis of information richness (high information density), it also takes into account topic concentration (large topic variation), so as to better guide the document segmentation process and improve the effect of subsequent retrieval and generation tasks.

[0113] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0114] The present invention also provides a document segmentation system based on adaptive slicing of hierarchical titles in large model retrieval enhancement generation, and the specific technical solution is as follows:

[0115] The acquisition module is used to: acquire the document to be segmented, and segment the document to be segmented according to the title type to obtain at least one set of original segments;

[0116] The calculation module is used to calculate the optimal number of cuts of any original cut block according to the information density and the theme change degree corresponding to the original cut block;

[0117] The segmentation module is used to segment the original block according to the optimal segmentation quantity.

[0118] It should be noted that the beneficial effects of the document segmentation system with adaptive slicing generated by large model retrieval enhancement provided in the above embodiment are the same as the beneficial effects of the document segmentation method with adaptive slicing generated by large model retrieval enhancement provided in the above embodiment, which will not be repeated here. In addition, when the system provided in the above embodiment realizes its functions, it only takes the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0119] like Figure 2As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above methods, specifically:

[0120] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements a document segmentation method for adaptive slicing generated by a large model retrieval enhancement provided in the above embodiment. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 300 may also include other components for implementing device functions, which will not be described in detail here.

[0121] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any one of the above methods.

[0122] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0123] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above methods.

[0124] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0125] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0126] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0127] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A document segmentation method based on adaptive slicing of hierarchical titles in large model retrieval enhancement generation, characterized in that: include: Obtaining a document to be segmented, and segmenting the document to be segmented according to the title type to obtain at least one set of original segments; According to the information density and the degree of topic change corresponding to any original block, the optimal number of divisions of the original block is calculated; The original block is segmented according to the optimal number of segments.

2. According to claim 1, the document segmentation method based on adaptive slicing of hierarchical titles in large model retrieval enhancement generation is characterized in that: The information density is calculated as follows: Among them, T i represents the information density corresponding to the i-th original block, N d is the total number of words after deduplication in the i-th original block, t i N d The weight of the i-th word in the deduplicated text obtained by the TF-IDF algorithm, n i Represents N d The frequency of occurrence of the i-th word in the deduplicated text, N all Represents the number of all words in the i-th original block.

3. According to claim 2, the document segmentation method based on adaptive slicing of hierarchical titles in large model retrieval enhancement generation is characterized in that: The calculation method of the topic change degree is: D i =D JS (v i-1 ,v i ); Among them, D i is the topic change degree corresponding to the i-th original block, D JS is the Jensen-Shannon divergence, v i-1 represents the vector distribution corresponding to the i-1th original cut, v i Represents the vector distribution of the current i-th original slice.

4. According to claim 3, the document segmentation method based on adaptive slicing of hierarchical titles in large model retrieval enhancement generation is characterized in that: The optimal number of cuts is calculated as follows: Among them, S i is the optimal number of segments corresponding to the ith original segment, C represents the total number of characters corresponding to the ith original segment, and α, β, and γ are adjustment parameters.

5. A document segmentation system based on adaptive slicing of hierarchical titles in large-scale model retrieval enhancement generation, characterized in that: include: The acquisition module is used to: acquire the document to be segmented, and segment the document to be segmented according to the title type to obtain at least one set of original segments; The calculation module is used to calculate the optimal number of cuts of any original cut block according to the information density and the theme change degree corresponding to the original cut block; The segmentation module is used to segment the original block according to the optimal segmentation quantity.

6. According to claim 5, the document segmentation system based on adaptive slicing of hierarchical titles in large model retrieval enhancement generation is characterized in that: The information density is calculated as follows: Among them, T i represents the information density corresponding to the i-th original block, N d is the total number of words after deduplication in the i-th original block, t i N d The weight of the i-th word in the deduplicated text obtained by the TF-IDF algorithm, n i Represents N d The frequency of occurrence of the i-th word in the deduplicated text, N all Represents the number of all words in the i-th original block.

7. According to claim 6, the document segmentation system based on adaptive slicing of hierarchical titles in large model retrieval enhancement generation is characterized in that: The calculation method of the topic change degree is: D i =D JS (v i-1 ,v i ); Among them, D i is the topic change degree corresponding to the i-th original block, D JS is the Jensen-Shannon divergence, v i-1 represents the vector distribution corresponding to the i-1th original cut, v i Represents the vector distribution of the current i-th original slice.

8. According to claim 7, the document segmentation system based on adaptive slicing of hierarchical titles in large model retrieval enhancement generation is characterized in that: The optimal number of cuts is calculated as follows: Among them, S i is the optimal number of segments corresponding to the ith original segment, C represents the total number of characters corresponding to the ith original segment, and α, β, and γ are adjustment parameters.

9. An electronic device, characterized in that: The electronic device comprises a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements the method according to any one of claims 1 to 4.

Images