Document segmentation method for large model retrieval enhancement generation

Abstract

The invention discloses a document segmentation method for large model retrieval enhancement generation, and relates to the technical field of document segmentation, the method comprises the following steps: taking any secondary title and text content under the secondary title as a to-be-analyzed object, and counting the number of characters in each to-be-analyzed object, when the number of the characters exceeds the preset number of the characters, paragraph splitting is carried out on the text content; and carrying out segmentation confidence calculation on each split part, and carrying out merging or segmentation processing on the adjacent parts based on the segmentation confidence corresponding to the adjacent parts. According to the method, the segmentation confidence coefficient of the adjacent parts is calculated, the document can be accurately and partially segmented, the segmentation effect is effectively guaranteed, and in addition, based on the segmentation confidence coefficient, the semantic integrity after segmentation and before and after segmentation can be effectively guaranteed.

Description

Background Art

[0002] Retrieval Augmented Generation is a technology applied in the field of knowledge content retrieval. It combines vector databases and large language models, and aims to enhance the generation capabilities of large models through retrieval technology. The construction process of the vector database is: document preparation → document segmentation → slice vectorization → storage. Document segmentation is to optimize the construction of the vector database and improve the generation quality of the large language model. In this technology, the large model will provide reference answers based on the document segmentation content after retrieval sorting, so document segmentation is particularly important for the answer quality of the large model.

[0003] In terms of document segmentation technology in retrieval enhanced generation, previous methods mainly include two methods: segmentation based on a fixed number of words and segmentation based on specified punctuation marks.

[0004] The method based on fixed word count segmentation often cannot adapt well to the changes in content and structure of different documents. Since the paragraph length and content of different documents vary greatly, fixed word count segmentation may lead to inaccurate segmentation results, affecting subsequent document processing and analysis.

[0005] Although segmentation based on specified punctuation marks solves the segmentation accuracy problem to a certain extent, it still has limitations. Specific punctuation marks may appear at different frequencies and positions in different documents, which affects the accuracy of the segmentation results. Summary of the invention

[0006] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a document segmentation method for large model retrieval enhancement generation, which is as follows:

[0007] 1) In the first aspect, the present invention provides a document segmentation method for large model retrieval enhancement generation, and the specific technical solution is as follows:

[0008] Taking any secondary title and the text content under the secondary title as an object to be analyzed, counting the number of characters in each object to be analyzed, and dividing the text content into paragraphs when the number of characters exceeds a preset number of characters;

[0009] The segmentation confidence of each split part is calculated, and the adjacent parts are merged or split based on the segmentation confidence corresponding to the adjacent parts.

[0010] The beneficial effects of the document segmentation method for large model retrieval enhancement generation provided by the present invention are as follows:

[0011] By calculating the segmentation confidence of adjacent parts, the document can be partially segmented accurately, which effectively guarantees the segmentation effect. In addition, based on the segmentation confidence, the integrity of the semantics before and after segmentation can be effectively guaranteed.

[0012] Based on the above solution, the present invention can also be improved as follows.

[0013] Furthermore, the process of splitting the text content into paragraphs is specifically as follows:

[0014] The paragraphs are split according to the line breaks, and in each split paragraph, the sentences are split according to the periods. Each sentence obtained by the split is used as the split part to calculate the segmentation confidence.

[0015] Furthermore, the process of calculating the segmentation confidence of each part after the split is specifically as follows: starting from the second split part, data cleaning is performed on the current part and the part between the current part and the secondary title, and after data cleaning, the confidence scores of the text vectors corresponding to the current part and the part between the current part and the secondary title are calculated, and the confidence scores are used as the segmentation confidence.

[0016] Furthermore, the data cleaning specifically includes: removing numbers and removing punctuation marks.

[0017] Furthermore, the specific calculation method of the confidence score is:

[0018] It is obtained by weighting the cosine similarity score and the correlation score;

[0019] The cosine similarity score is the cosine similarity score of the space vector between the current part and the previous part;

[0020] The relevance score is a relevance score between the current part and the previous part.

[0021] 2) In the second aspect, the present invention also provides a document segmentation system for large model retrieval enhancement generation, and the specific technical solution is as follows:

[0022] The splitting module is used to: take any secondary title and the text content under the secondary title as an object to be analyzed, count the number of characters in each object to be analyzed, and split the text content into paragraphs when the number of characters exceeds a preset number of characters;

[0023] The processing module is used to calculate the segmentation confidence of each split part, and merge or split the adjacent parts based on the segmentation confidence corresponding to the adjacent parts.

[0024] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, wherein the processor is coupled to a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above methods.

[0025] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor so that a computer implements any of the above methods.

[0026] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0028] Figure 1 A schematic diagram of a flow chart of a document segmentation method for large model retrieval enhanced generation according to an embodiment of the present invention;

[0029] Figure 2 The figure is a structural framework diagram of an electronic device. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0031] like Figure 1 As shown, a document segmentation method for large model retrieval enhanced generation according to an embodiment of the present invention includes the following steps:

[0032] S1, taking any secondary title and the text content under the secondary title as an object to be analyzed, counting the number of characters in each object to be analyzed, and dividing the text content into paragraphs when the number of characters exceeds a preset number of characters;

[0033] S2, calculating the segmentation confidence of each split part, and merging or splitting the adjacent parts based on the segmentation confidence corresponding to the adjacent parts.

[0034] The beneficial effects of the document segmentation method for large model retrieval enhancement generation provided by the present invention are as follows:

[0035] By calculating the segmentation confidence of adjacent parts, the document can be partially segmented accurately, which effectively guarantees the segmentation effect. In addition, based on the segmentation confidence, the integrity of the semantics before and after segmentation can be effectively guaranteed.

[0036] The second-level title is a title with the same serial number as the main text, occupying a separate line, and without punctuation at the end. The title is a short sentence that indicates the content of an article or work, and is generally divided into a general title, subtitle, and sub-title. The first-level title, second-level title, and subsequent titles all have the same serial number and different font sizes.

[0037] The preset number of characters is a manually set value and is not specifically limited here.

[0038] In the process of segmentation confidence, it is first determined whether there are typos in the current part (current sentence), and if the determination result is yes, the typos are corrected. At the same time, it is determined whether the meanings expressed by the corrected current sentence are similar to those expressed by the current sentence before correction, and when the initial similarity between the current sentences before and after correction exceeds a threshold, it is determined that the meanings expressed by the current sentences before and after correction are the same. It should be noted that the threshold is usually set to more than 80%, and the specific value is adjusted according to the actual situation, and is not specifically limited here.

[0039] When the similarity does not exceed the threshold, the keywords corresponding to the sentences before and after the correction are extracted, and the keywords are compared and analyzed to determine whether the sentences before and after the correction are sentences with opposite meanings. If so, a manual check is notified. If not, no correction is made for the typos. Opposite meanings refer to the conversion between affirmation and negation or antonyms.

[0040] The calculation process of segmentation confidence can be divided into the following two types:

[0041] The first method is to extract at least two keywords from the corrected current sentence after the typos are corrected, and extract at least two keywords from the adjacent previous sentence, perform semantic analysis on the extracted keywords, and determine the similarity and correlation between the keywords corresponding to the corrected current sentence and the keywords corresponding to the adjacent previous sentence; perform word segmentation on the corrected current sentence and the adjacent previous sentence, determine the weight corresponding to each word segment in the history library according to the word segmentation corresponding to each sentence, and determine the score corresponding to each sentence based on the weight corresponding to each word segmentation, and determine the weight corresponding to the score of each sentence in combination with the score of each sentence, and perform weighted summation on the similarity and correlation based on the above two weights, that is, perform weighted summation on the similarity and the above two weights to obtain the total similarity, and perform weighted summation on the correlation and the above two weights to obtain the total correlation, and determine the segmentation confidence corresponding to the two sentences based on the weights corresponding to the similarity and the correlation.

[0042] The second method is to use BERT to generate space vectors x and y for the current sentence and all the rows accumulated from the last segmentation point to the current sentence in this paragraph, and calculate the cosine similarity between the two.

[0043] emb_sim represents two points x(x1,x2,…,x n ), y(y1,y2,…,y n ) is the cosine similarity score between them.

[0044]

[0045] Calculate the relevance score (nsp_score) between the current sentence and the previous sentence.

[0046] BERT is an open source pre-trained semantic model that is pre-trained for context judgment tasks and can determine whether there is context between two sentences.

[0047] In this task, two adjacent sentences of input text are separated by [SEP]. [CLS] is added at the beginning of the first sentence and [SEP] is added at the end of the second sentence.

[0048] For example: [CLS] We conducted experiments based on existing open source datasets [SEP] The experimental results prove that our algorithm is advanced [SEP]

[0049] nsp_score=seq_relationship_score(C,S)

[0050] Where C represents the current sentence and S represents the sentence before the current sentence.

[0051] seq_relationship_score is the output of a fully connected layer in bert followed by a softmax layer.

[0052] Then, the scores of the two are multiplied by their respective weights and added together to obtain the final segmentation confidence score of the current sentence. Considering that the importance of the overall similarity of the previous text is greater than the adjacent correlation of the previous sentence, the weights of the two are 0.6 and 0.4 respectively.

[0053] cf_score=0.6*emb_sim+0.4*nsp_score

[0054] If the confidence score cf_score>0.5, the current sentence and the previous sentence are recorded as segmentation points, and the part between the current sentence and the previous segmentation point is recorded as a complete text block.

[0055] The semantic integrity is ensured by calculating the similarity between the current sentence and all the sentences accumulated from the last segmentation point to the current line in this paragraph, and the correlation is ensured by calculating the adjacent correlation between the current sentence and the previous sentence. In this way, the document can be accurately segmented into blocks, the segmentation effect is effectively guaranteed, and the answer quality of the large model is further improved by improving the quality of the reference document.

[0056] Furthermore, the process of splitting the text content into paragraphs is specifically as follows:

[0057] The paragraphs are split according to the line breaks, and in each split paragraph, the sentences are split according to the periods. Each sentence obtained by the split is used as the split part to calculate the segmentation confidence.

[0058] Furthermore, the process of calculating the segmentation confidence of each part after the split is specifically as follows: starting from the second split part, data cleaning is performed on the current part and the part between the current part and the secondary title, and after data cleaning, the confidence scores of the text vectors corresponding to the current part and the part between the current part and the secondary title are calculated, and the confidence scores are used as the segmentation confidence.

[0059] Furthermore, the data cleaning specifically includes: removing numbers and removing punctuation marks.

[0060] Furthermore, the specific calculation method of the confidence score is:

[0061] It is obtained by weighting the cosine similarity score and the correlation score;

[0062] The cosine similarity score is the cosine similarity score of the space vector between the current part and the previous part;

[0063] The relevance score is a relevance score between the current part and the previous part.

[0064] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0065] The present invention also provides a document segmentation system for large model retrieval enhancement generation, and the specific technical solution is as follows:

[0066] The splitting module is used to: take any secondary title and the text content under the secondary title as an object to be analyzed, count the number of characters in each object to be analyzed, and split the text content into paragraphs when the number of characters exceeds a preset number of characters;

[0067] The processing module is used to calculate the segmentation confidence of each split part, and merge or split the adjacent parts based on the segmentation confidence corresponding to the adjacent parts.

[0068] It should be noted that the beneficial effects of the document segmentation system for large model retrieval enhancement generation provided by the above embodiment are the same as the beneficial effects of the document segmentation method for large model retrieval enhancement generation provided by the above embodiment, which will not be repeated here. In addition, when the system provided by the above embodiment realizes its functions, it only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system is divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0069] like Figure 2 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above methods, specifically:

[0070] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements a document segmentation method for large model retrieval enhancement generation provided by the above embodiment. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the electronic device 300 may also include other components for implementing device functions, which will not be repeated here.

[0071] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any one of the above methods.

[0072] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0073] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above methods.

[0074] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0075] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0076] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0077] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A document segmentation method for large model retrieval enhancement generation, characterized in that: include: Taking any secondary title and the text content under the secondary title as an object to be analyzed, counting the number of characters in each object to be analyzed, and dividing the text content into paragraphs when the number of characters exceeds a preset number of characters; The segmentation confidence of each split part is calculated, and the adjacent parts are merged or split based on the segmentation confidence corresponding to the adjacent parts.

2. According to claim 1, a document segmentation method for large model retrieval enhancement generation is characterized in that: The process of splitting the text content into paragraphs is specifically as follows: The paragraphs are split according to the line breaks, and in each split paragraph, the sentences are split according to the periods. Each sentence obtained by the split is used as the split part to calculate the segmentation confidence.

3. According to claim 2, a document segmentation method for large model retrieval enhancement generation is characterized in that: The process of calculating the segmentation confidence of each part after the split is specifically as follows: starting from the second split part, data cleaning is performed on the current part and the part between the current part and the secondary title, and after data cleaning, the confidence scores of the text vectors corresponding to the current part and the part between the current part and the secondary title are calculated, and the confidence scores are used as the segmentation confidence.

4. According to claim 3, a document segmentation method for large model retrieval enhancement generation is characterized in that: The data cleaning specifically includes: removing numbers and removing punctuation marks.

5. According to claim 3, a document segmentation method for large model retrieval enhancement generation is characterized in that: The specific calculation method of the confidence score is: It is obtained by weighting the cosine similarity score and the correlation score; The cosine similarity score is the cosine similarity score of the space vector between the current part and the previous part; The relevance score is a relevance score between the current part and the previous part.

6. A document segmentation system for large model retrieval enhancement generation, characterized in that: include: The splitting module is used to: take any secondary title and the text content under the secondary title as an object to be analyzed, count the number of characters in each object to be analyzed, and split the text content into paragraphs when the number of characters exceeds a preset number of characters; The processing module is used to calculate the segmentation confidence of each split part, and merge or split the adjacent parts based on the segmentation confidence corresponding to the adjacent parts.

7. A document segmentation system for large model retrieval enhancement generation according to claim 6, characterized in that: The process of splitting the text content into paragraphs is specifically as follows: The paragraphs are split according to the line breaks, and in each split paragraph, the sentences are split according to the periods. Each sentence obtained by the split is used as the split part to calculate the segmentation confidence.

8. A document segmentation system for large model retrieval enhancement generation according to claim 7, characterized in that: The process of calculating the segmentation confidence of each part after the split is specifically as follows: starting from the second split part, data cleaning is performed on the current part and the part between the current part and the secondary title, and after data cleaning, the confidence scores of the text vectors corresponding to the current part and the part between the current part and the secondary title are calculated, and the confidence scores are used as the segmentation confidence.

9. An electronic device, characterized in that: The electronic device comprises a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements the method according to any one of claims 1 to 5.

Images