A document segmentation method of adaptive slice of large model retrieval enhanced generation

Abstract

The application discloses a document segmentation method for adaptive segmentation of large model retrieval enhanced generation, and relates to the technical field of large model retrieval enhanced generation, which comprises the following steps: obtaining a document to be segmented, and segmenting the document to be segmented according to a title type to obtain at least one group of original cut blocks; calculating an optimal segmentation number of any original cut block according to information density and theme variation degree corresponding to the original cut block; and segmenting the original cut block according to the optimal segmentation number. According to the application, the document is first segmented according to hierarchical titles, then the information density and the theme variation degree under the hierarchical titles are calculated, the optimal segmentation size under the hierarchical titles is automatically calculated in the unit of the hierarchical titles, and the adaptive segmentation of the document is guided, so that the effect of subsequent retrieval and generation tasks is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of large-model retrieval enhancement generation, and in particular to a document segmentation method for adaptive slicing of large-model retrieval enhancement generation. Background Art

[0002] Retrieval Augmented Generation (RAG) is a technology applied to knowledge content retrieval. It combines a vector database with a large language model, aiming to enhance the generation capabilities of the large model through retrieval technology. The vector database construction process is as follows: document preparation → document segmentation → slice vectorization → storage. Document segmentation optimizes the construction of the vector database and improves the generation quality of the large language model. In this technology, the large model provides reference answers based on the segmented content of the retrieved and ranked documents, making document segmentation particularly important for the quality of the large model's answers.

[0003] In terms of document segmentation technology in retrieval-enhanced generation, previous methods mainly include two methods: segmentation based on a fixed number of words and segmentation based on specified punctuation marks.

[0004] Methods based on fixed word counts often fail to adapt well to changes in document content and structure. Because paragraph length and content vary significantly across documents, fixed word counts can lead to inaccurate segmentation results, impacting subsequent document processing and analysis.

[0005] Given the limitations of traditional document segmentation technology, further optimization and improvement are needed to meet higher requirements for document segmentation technology. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and specifically provide a document segmentation method for adaptive slicing based on large-scale model retrieval enhancement generation, as follows:

[0007] 1) In the first aspect, the present invention provides a document segmentation method for adaptive slicing generated by large-scale model retrieval enhancement. The specific technical solution is as follows:

[0008] Obtaining a document to be segmented, and segmenting the document to be segmented according to title type to obtain at least one set of original segments;

[0009] Calculate the optimal number of cuts for any original cut based on the information density and topic variation of the original cut;

[0010] The original block is segmented according to the optimal segmentation quantity.

[0011] The beneficial effects of the document segmentation method for adaptive slicing with large-scale model retrieval enhancement generation provided by the present invention are as follows:

[0012] First, the document is segmented according to the hierarchical title, and then the information density and topic change degree under the hierarchical title are calculated. The optimal segmentation size under the hierarchical title is automatically calculated using the hierarchical title as the unit to guide the adaptive segmentation of the document, so as to improve the effect of subsequent retrieval and generation tasks.

[0013] Based on the above solution, the present invention can also be improved as follows.

[0014] Furthermore, the information density is calculated as follows:

[0015]

[0016] Among them, T i Represents the information density corresponding to the i-th original block, N d is the total number of characters in the original i-th block after deduplication, t i Indicates N d The weight of the i-th word in the deduplicated text obtained by the TF-IDF algorithm, n i Represents N d The frequency of occurrence of the i-th word in the deduplicated text, N all Represents the number of all words in the i-th original block.

[0017] Furthermore, the topic change degree is calculated as follows:

[0018] D i =D JS (v i-1 ,v i );

[0019] Among them, D i is the topic change degree corresponding to the i-th original block, D JS is the Jensen-Shannon divergence, v i-1 represents the vector distribution corresponding to the i-1th original cut, v i Represents the vector distribution of the current i-th original slice.

[0020] Furthermore, the optimal number of splits is calculated as follows:

[0021]

[0022] Among them, S i is the optimal number of segments corresponding to the ith original segment, C represents the total number of characters corresponding to the ith original segment, and α, β, and γ are adjustment parameters.

[0023] 2) In a second aspect, the present invention further provides a document segmentation system based on adaptive slicing of hierarchical titles in large-scale model retrieval enhancement generation. The specific technical solution is as follows:

[0024] The acquisition module is used to: acquire a document to be segmented, and segment the document to be segmented according to the title type to obtain at least one set of original segments;

[0025] The calculation module is used to calculate the optimal number of cuts for any original cut according to the information density and the degree of topic change corresponding to the original cut;

[0026] The segmentation module is used to segment the original block according to the optimal segmentation quantity.

[0027] Based on the above solution, the present invention can also be improved as follows.

[0028] Furthermore, the information density is calculated as follows:

[0029]

[0030] Among them, T i Represents the information density corresponding to the i-th original block, N d is the total number of characters in the original i-th block after deduplication, t i Indicates N d The weight of the i-th word in the deduplicated text obtained by the TF-IDF algorithm, n i Represents N d The frequency of occurrence of the i-th word in the deduplicated text, N all Represents the number of all words in the i-th original block.

[0031] Furthermore, the topic change degree is calculated as follows:

[0032] D i =D JS (v i-1 ,v i );

[0033] Among them, D i is the topic change degree corresponding to the i-th original block, D JS is the Jensen-Shannon divergence, v i-1 represents the vector distribution corresponding to the i-1th original cut, v i Represents the vector distribution of the current i-th original slice.

[0034] Furthermore, the optimal number of splits is calculated as follows:

[0035]

[0036] Among them, S i is the optimal number of segments corresponding to the ith original segment, C represents the total number of characters corresponding to the ith original segment, and α, β, and γ are adjustment parameters.

[0037] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, wherein the processor is coupled to a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above methods.

[0038] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable a computer to implement any of the above methods.

[0039] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0041] Figure 1 A flowchart of a document segmentation method for adaptive slicing generated by large-scale model retrieval enhancement according to an embodiment of the present invention;

[0042] Figure 2 This is a structural framework diagram of an electronic device. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0044] like Figure 1 As shown, a document segmentation method for adaptive slicing based on large-scale model retrieval enhancement generation according to an embodiment of the present invention includes the following steps:

[0045] S1, obtaining a document to be segmented, and segmenting the document according to title type to obtain at least one set of original segments;

[0046] S2, calculate the optimal number of cuts for any original cut according to the information density and topic variation of the original cut;

[0047] S3, dividing the original block into pieces according to the optimal number of cuts.

[0048] The beneficial effects of the document segmentation method for adaptive slicing with large-scale model retrieval enhancement generation provided by the present invention are as follows:

[0049] First, the document is segmented according to the hierarchical title, and then the information density and topic change degree under the hierarchical title are calculated. The optimal segmentation size under the hierarchical title is automatically calculated using the hierarchical title as the unit to guide the adaptive segmentation of the document, so as to improve the effect of subsequent retrieval and generation tasks.

[0050] The title type refers to:

[0051] Simple title: This type of title usually uses only one word or phrase to express the subject and is suitable for internal documents or simple reports, such as "Notice", "Request", "Report", etc.

[0052] Complete title: Use a complete sentence or phrase to express the subject, suitable for external documents or formal reports, such as "Report on XXX Matter", "Investigation Report on XXX Issue", etc.

[0053] Prefix title: Add a prefix before the simple title or full title to express the subject more accurately, such as "Urgent Notice", "Important Report", "Recommendation Report on XXX Issue", etc.

[0054] Standard title: This title consists of three elements: the issuing authority, the reason, and the name of the document type. It is usually used in formal documents, such as "Notice on the Implementation of XXX Activities".

[0055] Binary title: It consists of the issuing agency and the name of the document type, or the reason and the name of the document type. It is suitable for relatively concise official document titles.

[0056] News-style titles: Mainly used in briefings, investigative reports, speeches and other types of documents. They are very casual and the text is lively. Some have only one title, while others have two titles, such as "XXX event ended successfully" and "XXX unit 2023 annual work summary".

[0057] First-level headings are numbered "I.", use size 4 boldface, and appear on their own line with no punctuation at the end. A first-level heading is usually the largest heading in an article and summarizes the main content or chapters.

[0058] Secondary headings are numbered "(I)" and are written in the same font size as the main text. They are placed on their own line and have no punctuation at the end. Secondary headings are usually sub-arguments under the primary heading, further subdividing the content.

[0059] The optimal number of splits is calculated as follows:

[0060]

[0061] The adjustment parameters α, β, and γ are determined as follows:

[0062] Determine the first information density of the document to be segmented, and judge whether the first information density is lower than the preset minimum information density. If so, further judge whether the current original segment also includes the previous original segment. If so, extract the first keyword of the title of the current original segment and the second keyword of the current original segment, and extract the third keyword of the title of the previous original segment and the fourth keyword of the previous original segment, determine the first correlation and the first similarity between the first keyword and the third keyword, determine the second correlation and the second similarity between the second keyword and the fourth keyword, determine a first similarity score based on the first correlation and the first similarity, determine a second similarity score based on the second correlation and the second similarity, determine a first total score of the current original segment based on the first similarity score and the second similarity score, and determine the range of α based on the first total score.

[0063] The process of determining the similarity score is as follows:

[0064] The similarity score is determined by determining the weight corresponding to the correlation degree and the similarity.

[0065] The process of determining the first total score of the current original slice based on the first similarity score and the second similarity score is as follows:

[0066] Determine the first overall score based on the title weight and the text weight.

[0067] If the first information density is not lower than the preset minimum information density, or if the current original segment does not include the previous original segment, the preset α is used as α in the optimal segmentation number of the current original segment, and whether β and γ need to be adjusted is further determined.

[0068] The preceding original segment refers to an original segment that precedes the current original segment among a plurality of original segments arranged in the order of the text of the document to be segmented.

[0069] If any original segment contains more than a preset number of professional terms and a text description of technical details that exceeds a preset length, a target number of professional terms exceeding a predetermined number and a target length exceeding a predetermined length of the text description are determined. Based on a table corresponding to the number of professional terms exceeding a predetermined number and a table corresponding to the length exceeding a predetermined number and a second target weight corresponding to the target length exceeding a predetermined number, a second total score corresponding to the original segment is determined based on the first and second target weights, the number of professional terms, and the text description. The range of β is determined based on the second total score.

[0070] If any original segment does not contain more than a preset number of professional terms, or does not contain a text description of technical details exceeding a preset length, the preset β is used as the β in the optimal number of segments for the current original segment. It is then determined whether γ needs to be adjusted.

[0071] Based on the third and fourth keywords, it is determined whether the current original segment and the previous original segment are completely different topics. In other words, it is determined whether the overlap between the third and fourth keywords is lower than the preset overlap. If so, γ is adjusted. If not, the preset γ is used as the γ in the optimal number of segments for the current original segment.

[0072] Furthermore, the information density is calculated as follows:

[0073]

[0074] Among them, T i Represents the information density corresponding to the i-th original block, N d is the total number of characters in the original i-th block after deduplication, t i Indicates N d The weight of the i-th word in the deduplicated text obtained by the TF-IDF algorithm, n i Represents N d The frequency of occurrence of the i-th word in the deduplicated text, N all Represents the number of all words in the i-th original block.

[0075] Furthermore, the topic change degree is calculated as follows:

[0076] D i =D JS (v i-1 ,v i );

[0077] Among them, D i is the topic change degree corresponding to the i-th original block, D JSis the Jensen-Shannon divergence, v i-1 represents the vector distribution corresponding to the i-1th original cut, v i Represents the vector distribution of the current i-th original slice.

[0078] Furthermore, the optimal number of splits is calculated as follows:

[0079]

[0080] Among them, S i is the optimal number of segments corresponding to the ith original segment, C represents the total number of characters corresponding to the ith original segment, and α, β, and γ are adjustment parameters.

[0081] 1. Document preparation: Collect the documents that need to be processed.

[0082] Preprocessing: Split each document into hierarchical titles (also known as sub-titles, interjections, and sub-titles).

[0083] Adaptive slicing:

[0084] Calculation formula:

[0085]

[0086] in:

[0087] S i : The optimal slice size of the i-th level title;

[0088] C: constant, representing the total number of characters in the text under the i-th level title;

[0089] α: Adjustment parameter used to adjust the basic slice size. When there is no obvious difference in information density in the document, this parameter determines the default slice size;

[0090] If the document has low overall information density and little variation in topics, then α will play a dominant role and provide a reasonable default slice size. For example, for a narrative essay, since its information density is relatively uniform, α determines the slice size of most paragraphs.

[0091] β: Adjustment parameter used to weight information density T i If β is larger, it means that the information density has a greater impact on the slice size;

[0092] For example, in a technical document, some paragraphs may contain a large amount of terminology and technical details (high information density). In this case, a larger β will make the slices of these paragraphs smaller in order to better capture important information.

[0093] γ: Adjustment parameter for weighted topic change score D i If γ is larger, it means that the influence of the theme change on the slice size is greater;

[0094] If a document suddenly turns from discussing one technology to a completely different topic, increasing γ will cause the paragraphs near these turning points to be cut into smaller slices to maintain the thematic coherence of each slice.

[0095] T i : Information density of the i-th level title. Calculated based on TF-IDF (term frequency-inverse document frequency). Paragraphs with higher information density usually contain more key information, so smaller slices may be needed to ensure that information is not overly dispersed.

[0096] D i : The topic change degree of the i-th level title. Calculated using the Jensen-Shannon divergence, it measures the degree of topic shift between one level title and the next. A high topic change score means that the paragraph may be a transition between two different topics and should be segmented more finely to maintain thematic coherence.

[0097] Specifically, T i Represents the information density of the title text at the i-th level, ranging from 0 to 1, and the formula is:

[0098]

[0099] in:

[0100] N d Represents the total number of all words in the input text after deduplication;

[0101] t i Represents this N d The weight of the i-th word among the deduplicated words is obtained by the TF-IDF algorithm;

[0102] n i Represents the frequency of the word appearing;

[0103] N all Represents the number of all words in the paragraph;

[0104] D i Represents the topic change degree of the i-th level title, ranging from 0 to 1, and the formula is:

[0105] D i =D JS (v i-1 ,v i )

[0106] in:

[0107] D JS is the Jensen-Shannon divergence, which measures the difference between two probability distributions;

[0108] v i-1 Represents the vector distribution of the previous paragraph (i-1);

[0109] v i Represents the vector distribution of the current i-th paragraph of text;

[0110] The vector is generated by the general BERT model;

[0111] Note that when i is the first segment, D i =0.

[0112] The present invention can segment each paragraph, and on the basis of information richness (high information density), it also takes into account topic concentration (large topic variation), thereby better guiding the document segmentation process and improving the effects of subsequent retrieval and generation tasks.

[0113] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the scope of protection of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0114] The present invention also provides a document segmentation system based on adaptive slicing of hierarchical titles in large-scale model retrieval enhancement generation. The specific technical solution is as follows:

[0115] The acquisition module is used to: acquire a document to be segmented, and segment the document to be segmented according to the title type to obtain at least one set of original segments;

[0116] The calculation module is used to calculate the optimal number of cuts for any original cut according to the information density and the degree of topic change corresponding to the original cut;

[0117] The segmentation module is used to segment the original block according to the optimal segmentation quantity.

[0118] It should be noted that the beneficial effects of the document segmentation system with adaptive slicing generated by large-scale retrieval enhancement provided by the above embodiment are the same as the beneficial effects of the document segmentation method with adaptive slicing generated by large-scale retrieval enhancement provided by the above embodiment, and will not be repeated here. In addition, when the system provided by the above embodiment realizes its functions, it only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above embodiment belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0119] like Figure 2 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, which is coupled to a memory 310. The memory 310 stores at least one computer program 330. The at least one computer program 330 is loaded and executed by the processor 320 to enable the electronic device 300 to implement any of the above methods. Specifically:

[0120] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements a document segmentation method for adaptive slicing generated by a large model retrieval enhancement provided in the above embodiment. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 300 may also include other components for realizing device functions, which will not be described in detail here.

[0121] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable a computer to implement any of the above methods.

[0122] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0123] In an exemplary embodiment, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform any of the above methods.

[0124] It should be noted that the terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and to define a specific order or precedence. Where appropriate, the order used for similar objects may be interchanged, such that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0125] Those skilled in the art will appreciate that the present invention may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present invention may be implemented in the form of a computer program product embodied in one or more computer-readable media containing computer-readable program code.

[0126] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0127] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A document segmentation method based on adaptive slicing of hierarchical titles in large-scale model retrieval enhancement generation, characterized in that: include: Obtaining a document to be segmented, and segmenting the document to be segmented according to title type to obtain at least one set of original segments; Calculate the optimal number of cuts for any original cut based on the information density and topic variation of the original cut; Performing a segmentation process on the original block according to the optimal segmentation quantity; The information density is calculated as follows: Among them, T i Represents the information density corresponding to the i-th original block, N d is the total number of characters in the original i-th block after deduplication, t i Indicates N d The weight of the i-th word in the deduplicated text obtained by the TF-IDF algorithm, n i Represents N d The frequency of occurrence of the i-th word in the deduplicated text, N all Represents the number of all words in the i-th original block; The topic change degree is calculated as follows: D i =D JS (v i-1 ,v i ); Among them, D i is the topic change degree corresponding to the i-th original block, D JS is the Jensen-Shannon divergence, v i-1 represents the vector distribution corresponding to the i-1th original cut, v i Represents the vector distribution of the current i-th original slice; The optimal number of splits is calculated as follows: Among them, S i is the optimal number of segments corresponding to the ith original segment, C represents the total number of characters corresponding to the ith original segment, and α, β, and γ are adjustment parameters.

2. A document segmentation system based on adaptive slicing of hierarchical titles in large-scale model retrieval enhancement generation, adopting a document segmentation method based on adaptive slicing of hierarchical titles in large-scale model retrieval enhancement generation as claimed in claim 1, characterized in that: The system comprises: The acquisition module is used to: acquire a document to be segmented, and segment the document to be segmented according to the title type to obtain at least one set of original segments; The calculation module is used to calculate the optimal number of cuts for any original cut according to the information density and the degree of topic change corresponding to the original cut; The segmentation module is used to segment the original block according to the optimal segmentation quantity.

3. An electronic device, characterized in that: The electronic device includes a processor coupled to a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor, so that the electronic device implements the method according to claim 1.

4. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable the computer to implement the method according to claim 1.

Images