A document segmentation method for large model retrieval enhanced generation

Abstract

The application discloses a document segmentation method for large model retrieval enhanced generation, and relates to the technical field of document segmentation.The method comprises the following steps: taking any second-level title and text content under the second-level title as an object to be analyzed, counting the number of characters in each object to be analyzed, and carrying out paragraph splitting on the text content when the number of characters exceeds a preset number of characters; calculating the segmentation confidence of each part after splitting, and carrying out merging or segmentation processing on adjacent parts based on the segmentation confidence corresponding to the adjacent parts.The application can accurately segment the document by calculating the segmentation confidence of adjacent parts, effectively ensures the segmentation effect, and effectively ensures the integrity of the semantics before and after segmentation based on the segmentation confidence.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of document segmentation, in particular to a document segmentation method for retrieval augmented generation of large models. BACKGROUND

[0002] Retrieval augmented generation is a technology applied in the field of knowledge content retrieval, combining vector databases and large language models, aiming to enhance the generation capability of large models through retrieval technology. The construction process of the vector database is: document preparation → document segmentation → slice vectorization → storage. Document segmentation is to optimize the construction of the vector database and improve the generation quality of the large language model. In this technology, the large model will answer based on the retrieval sorted document segmentation content, so document segmentation is particularly important for the answer quality of the large model.

[0003] In the aspect of document segmentation technology in retrieval augmented generation, the previous methods mainly include two ways: fixed word number chunking and specified punctuation symbol chunking.

[0004] The method based on fixed word number chunking often cannot adapt well to the changes of different document contents and structures. Because of the large differences in paragraph length and content of different documents, fixed word number chunking may lead to inaccurate segmentation results, affecting subsequent document processing and analysis.

[0005] Although the specified punctuation symbol chunking solves the problem of segmentation accuracy to some extent, it still has limitations. The frequency and position of specific punctuation symbols may be different in different documents, affecting the accuracy of the chunking results. SUMMARY

[0006] The technical problem to be solved by the present application is to overcome the deficiencies of the prior art, and specifically provides a document segmentation method for retrieval augmented generation of large models, as follows:

[0007] 1) In a first aspect, the present application provides a document segmentation method for retrieval augmented generation of large models, and the specific technical solution is as follows:

[0008] Taking any second-level title and the text content under the second-level title as an object to be analyzed, counting the number of characters in each object to be analyzed, and when the number of characters exceeds a predetermined number of characters, performing paragraph splitting on the text content;

[0009] Calculating the segmentation confidence of each part after splitting, and based on the segmentation confidence corresponding to adjacent parts, performing merging or segmentation processing on the adjacent parts.

[0010] The beneficial effects of the document segmentation method for large-scale model retrieval enhancement generation provided by the present invention are as follows:

[0011] By calculating the segmentation confidence of adjacent parts, the document can be accurately segmented, effectively ensuring the segmentation effect. In addition, based on the segmentation confidence, the integrity of the semantics before and after segmentation can be effectively guaranteed.

[0012] Based on the above solution, the present invention can also be improved as follows.

[0013] Furthermore, the process of splitting the text content into paragraphs is specifically as follows:

[0014] The paragraphs are split according to the line breaks, and in each split paragraph, the sentences are split according to the periods. Each sentence obtained by the split is used as the split part to calculate the segmentation confidence.

[0015] Furthermore, the process of calculating the segmentation confidence of each part after the split is specifically as follows: starting from the second split part, data cleaning is performed on the current part and the part between the current part and the secondary title, and after data cleaning, the confidence scores of the text vectors corresponding to the current part and the part between the current part and the secondary title are calculated, and the confidence scores are used as the segmentation confidence.

[0016] Furthermore, the data cleaning specifically includes: removing numbers and removing punctuation marks.

[0017] Furthermore, the specific calculation method of the confidence score is:

[0018] Obtained through weighted processing of cosine similarity score and relevance score;

[0019] The cosine similarity score is the cosine similarity score of the space vector between the current part and the previous part;

[0020] The relevance score is a relevance score between the current portion and the previous portion.

[0021] 2) In a second aspect, the present invention further provides a document segmentation system for large-scale model retrieval and enhanced generation, the specific technical solution of which is as follows:

[0022] The splitting module is used to: take any secondary title and the text content under the secondary title as an object to be analyzed, count the number of characters in each object to be analyzed, and split the text content into paragraphs when the number of characters exceeds a preset number of characters;

[0023] The processing module is configured to: perform segmentation confidence calculation on each part after splitting, and perform merging or segmentation processing on adjacent parts based on the segmentation confidence corresponding to the adjacent parts.

[0024] 3) In a third aspect, the present application also provides an electronic device, which comprises a processor and a memory coupled to the processor, and the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to enable the electronic device to implement any of the above methods.

[0025] 4) In a fourth aspect, the present application also provides a computer readable storage medium, which stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable a computer to implement any of the above methods.

[0026] It should be noted that the technical solutions of the second to fourth aspects of the present application and the corresponding possible implementation manners have the beneficial effects as described above for the first aspect and its corresponding possible implementation manners, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0027] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the drawings:

[0028] Figure 1 A flowchart of a document segmentation method for large model retrieval enhanced generation according to an embodiment of the present application;

[0029] Figure 2 A structural framework diagram of an electronic device. DETAILED DESCRIPTION

[0030] To make the purposes, technical solutions and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the drawings.

[0031] As shown in the drawings, a document segmentation method for large model retrieval enhanced generation according to an embodiment of the present application comprises the following steps: Figure 1

[0032] S1, taking any secondary title and the text content under the secondary title as an object to be analyzed, counting the number of characters in each object to be analyzed, and when the number of characters exceeds a predetermined number of characters, performing paragraph splitting on the text content;

[0033] S2, performing segmentation confidence calculation on each part after splitting, and performing merging or segmentation processing on adjacent parts based on the segmentation confidence corresponding to the adjacent parts. ​

[0034] The document segmentation method for large model retrieval enhanced generation provided by the application has the following beneficial effects:

[0035] By calculating the segmentation confidence of adjacent parts, the document can be accurately segmented, and the segmentation effect is effectively guaranteed. In addition, based on the segmentation confidence, the semantic integrity before and after segmentation can be effectively guaranteed.

[0036] The secondary title is the same as the main title with the title number "one", occupies the line, and no punctuation mark is added at the end. The title is a short sentence that indicates the content of the article, work, etc. Generally, it is divided into general title, sub-title, and sub-title. The first title and the second title and the title below represent a serial number, and the font size is different.

[0037] The preset number of characters is a manually set value, which is not specifically limited here.

[0038] In the segmentation confidence process, first determine whether there is a wrong word in the current part (current sentence), and when the determination result is yes, correct the wrong word. At the same time, determine whether the meaning expressed by the corrected current sentence and the uncorrected current sentence is similar, and when the initial similarity corresponding to the current sentence before and after correction exceeds the threshold value, it is determined that the meaning expressed by the current sentence before and after correction is the same. It should be noted that the threshold value is usually more than 80%, and the specific value is adjusted according to the actual situation, which is not specifically limited here.

[0039] When the initial similarity does not exceed the threshold value, the keywords corresponding to the sentences before and after correction are extracted, and the keywords are compared and analyzed to determine whether the sentences before and after correction are opposite in meaning. If so, notify the artificial to check, if not, do not correct the wrong word. The opposite meaning refers to the conversion of affirmation and negation or antonyms, etc.

[0040] The calculation process of the segmentation confidence can be divided into the following two kinds:

[0041] The first kind, after the correction of the wrong word, at least two key words of the corrected current sentence are extracted, and at least two key words of the adjacent previous sentence are extracted, semantic analysis is performed on the extracted key words, and the similarity and correlation between the key words corresponding to the corrected current sentence and the key words corresponding to the adjacent previous sentence are determined; the corrected current sentence and the adjacent previous sentence are subjected to word segmentation processing, the weight corresponding to each word is determined in the history library according to the word segmentation corresponding to each sentence, and the score corresponding to each sentence is determined based on the weight corresponding to each word, the weight corresponding to the score of each sentence is determined in combination with the score of each sentence, and the similarity and correlation are weighted and summed based on the above two weights, that is, the similarity is weighted and summed with the above two weights to obtain the total similarity, the correlation is weighted and summed with the above two weights to obtain the total correlation, and the segmentation confidence corresponding to the two sentences is determined based on the weights corresponding to the similarity and correlation.

[0042] The second kind, using bert to generate space vectors x and y for the current sentence and all lines accumulated from the previous segmentation point to the current sentence in the paragraph respectively, and calculating the cosine similarity of the two.

[0043] emb_sim represents the cosine similarity score between two points x (x1, x2, …, x n ), y (y1, y2, …, y n ) in n-dimensional space.

[0044]

[0045] Calculate the relevance score (nsp_score) of the current sentence and the previous sentence.

[0046] Bert is an open source pre-trained semantic model, which pre-trains the context judgment task and can judge whether two sentences are in context.

[0047] In this task, the adjacent two sentences of the input text are separated by [SEP], the beginning of the first sentence is added with [CLS], and the end of the second sentence is added with [SEP].

[0048] For example: [CLS] we have carried out experiments based on existing open source data sets [SEP] experimental results prove that our algorithm has advancement [SEP]

[0049] nsp_score = seq_relationship_score (C, S)

[0050] Where C represents the current sentence, and S represents the previous sentence of the current sentence.

[0051] The seq_relationship_score is output by a full connection layer in the BERT and a softmax layer.

[0052] Then, the scores of the two are multiplied by respective weights, added, and the final segmentation confidence score of the current sentence is obtained. Considering that the overall similarity importance of the preceding text is greater than the adjacent relevance of the previous sentence, the weights of the two are 0.6 and 0.4, respectively.

[0053] The cf_score=0.6*emb_sim+0.4*nsp_score

[0054] If the confidence score cf_score>0.5, the current sentence and the previous sentence are recorded as a segmentation point, and are recorded. The part from the current sentence to the last segmentation point is recorded as a complete text block.

[0055] The similarity is calculated by the current sentence and all sentences accumulated from the last segmentation point to the current line in the paragraph to ensure semantic integrity, and the adjacent relevance is calculated by the current sentence and the previous sentence to ensure relevance. Thus, the document can be accurately segmented into blocks, effectively ensuring the segmentation effect, and the quality of the large model is further improved by improving the quality of the reference document.

[0056] Further, the process of segmenting the text content into paragraphs is specifically:

[0057] The paragraphs are segmented according to the line breaks, and in each segmented paragraph, the sentences are segmented according to the periods, and each segmented sentence is taken as a segmented part for calculating the segmentation confidence.

[0058] Further, the process of calculating the segmentation confidence of each segmented part is specifically: starting from the second segmented part, the current part and the part between the current part and the secondary title are data cleaned, and after data cleaning, the confidence score of the text vector corresponding to the current part and the part between the current part and the secondary title is calculated, and the confidence score is taken as the segmentation confidence.

[0059] Further, the data cleaning specifically includes: removing numbers and removing punctuation.

[0060] Further, the confidence score is specifically calculated as:

[0061] The cosine similarity score is obtained by weighting the cosine similarity score and the relevance score.

[0062] The cosine similarity score is the cosine similarity score of the space vector between the current part and the previous part.

[0063] The correlation score is a correlation score between the current part and the previous part.

[0064] In the above embodiments, although the steps are numbered S1, S2, etc., it is only a specific embodiment given by the present application, and those skilled in the art can adjust the execution order of S1, S2, etc. according to the actual situation, which is within the protection scope of the present application. It can be understood that in some embodiments, some or all of the above embodiments can be included.

[0065] The present application also provides a document segmentation system for large model retrieval enhanced generation, and the specific technical solutions are as follows:

[0066] The splitting module is configured to: take any second-level title and the text content under the second-level title as an analysis object, count the number of characters in each analysis object, and when the number of characters exceeds a preset number of characters, perform paragraph splitting on the text content;

[0067] The processing module is configured to: calculate the segmentation confidence of each part after splitting, and based on the segmentation confidence corresponding to adjacent parts, perform merging or segmentation processing on the adjacent parts.

[0068] It should be noted that the beneficial effects of the above embodiment of the document segmentation system for large model retrieval enhanced generation are the same as those of the above document segmentation method for large model retrieval enhanced generation, and will not be repeated here. In addition, when the system of the above embodiment implements its function, only the division of the above functional modules is exemplified, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the system is divided into different functional modules according to actual conditions to complete all or part of the above described functions. In addition, the system and method embodiments provided by the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0069] As shown in Figure 2 The electronic device 300 of the embodiment of the present application, as shown in the figure, includes a processor 320, the processor 320 is coupled with a memory 310, the memory 310 stores at least one computer program 330, the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 realizes any one of the above methods, specifically:

[0070] The electronic device 300 can have a large difference due to different configurations or performances, and can include one or more processors 320 (Central Processing Units, CPUs) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements the method for enhancing the document segmentation of the large model retrieval generation provided by the above-mentioned embodiments. Of course, the electronic device 300 can also have a wired or wireless network interface, a keyboard, an input and output interface and the like, so as to perform input and output, and the electronic device 300 can also include other components for realizing the functions of the device, which will not be described here.

[0071] The computer readable storage medium of the embodiment of the application stores at least one computer program, and the at least one computer program is loaded and executed by the processor, so that the computer implements any one of the above-mentioned methods.

[0072] Optionally, the computer readable storage medium can be a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a read-only compact disc (Compact Disc Read-Only Memory, CD-ROM), a magnetic tape, a floppy disk and an optical data storage device, etc.

[0073] In the exemplary embodiments, a computer program product or computer program is also provided, which includes computer instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the electronic device executes any one of the above-mentioned methods.

[0074] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application are used to distinguish similar objects, and represent a specific order or sequence. The order of use of similar objects can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described.

[0075] Those skilled in the art will appreciate that the present invention may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present invention may be implemented in the form of a computer program product embodied in one or more computer-readable media containing computer-readable program code.

[0076] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0077] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A document segmentation method for large model retrieval enhancement generation, characterized in that: include: Taking any secondary title and the text content under the secondary title as an object to be analyzed, counting the number of characters in each object to be analyzed, and splitting the text content into paragraphs when the number of characters exceeds a preset number of characters; Calculate the segmentation confidence of each split part, and merge or split the adjacent parts based on the segmentation confidence of the adjacent parts; The calculation process of segmentation confidence is either of the following two methods: The first method is to extract at least two keywords from the corrected current sentence after the typos are corrected, and extract at least two keywords from the adjacent previous sentence, perform semantic analysis on the extracted keywords, and determine the similarity and correlation between the keywords corresponding to the corrected current sentence and the keywords corresponding to the adjacent previous sentence; perform word segmentation on the corrected current sentence and the adjacent previous sentence, determine the weight corresponding to each word in the history library according to the word segmentation corresponding to each sentence, and determine the score corresponding to each sentence based on the weight corresponding to each word segmentation, and determine the weight corresponding to the score of each sentence based on the score of each sentence, and perform weighted summation on the similarity and correlation based on the above two weights. The process is: perform weighted summation on the similarity and the above two weights to obtain the total similarity, and perform weighted summation on the correlation and the above two weights to obtain the total correlation, and determine the segmentation confidence corresponding to the two sentences based on the weights corresponding to the total similarity and the total correlation; The second method is to use BERT to generate space vectors x and y for the current sentence and all the rows accumulated from the previous segmentation point to the current sentence in this paragraph, and calculate the cosine similarity score between the two. emb_sim represents the relationship between two points x(x1, x2, ..., x n ), y(y1,y2,…,y n ) between the cosine similarity scores; Calculate the relevance score between the current sentence and the previous sentence; cf_score=0.6*emb_sim+0.4*nsp_score Among them, cf_score is the segmentation confidence and nsp_score is the relevance score.

2. A document segmentation method for large model retrieval enhancement generation according to claim 1, characterized in that: The process of splitting the text content into paragraphs is specifically as follows: The paragraphs are split according to the line breaks, and in each split paragraph, the sentences are split according to the periods. Each sentence obtained by the split is used as the split part to calculate the segmentation confidence.

3. A document segmentation method for large model retrieval enhancement generation according to claim 2, characterized in that: The process of calculating the segmentation confidence of each part after the split is specifically as follows: starting from the second part split out, data cleaning is performed on the current part and the part between the current part and the secondary title, and after data cleaning, the confidence scores of the text vectors corresponding to the current part and the part between the current part and the secondary title are calculated, and the confidence scores are used as the segmentation confidence.

4. A document segmentation method for large model retrieval enhancement generation according to claim 3, characterized in that: The data cleaning specifically includes: removing numbers and removing punctuation marks.

5. A document segmentation system for large model retrieval enhancement generation, using the document segmentation method for large model retrieval enhancement generation according to claim 1, characterized in that: The system includes: The splitting module is used to: take any secondary title and the text content under the secondary title as an object to be analyzed, count the number of characters in each object to be analyzed, and split the text content into paragraphs when the number of characters exceeds a preset number of characters; The processing module is used to calculate the segmentation confidence of each split part, and merge or split the adjacent parts based on the segmentation confidence corresponding to the adjacent parts.

6. A document segmentation system for large model retrieval enhancement generation according to claim 5, characterized in that: The process of splitting the text content into paragraphs is specifically as follows: The paragraphs are split according to the line breaks, and in each split paragraph, the sentences are split according to the periods. Each sentence obtained by the split is used as the split part to calculate the segmentation confidence.

7. A document segmentation system for large model retrieval enhancement generation according to claim 6, characterized in that: The process of calculating the segmentation confidence of each part after the split is specifically as follows: starting from the second part split out, data cleaning is performed on the current part and the part between the current part and the secondary title, and after data cleaning, the confidence scores of the text vectors corresponding to the current part and the part between the current part and the secondary title are calculated, and the confidence scores are used as the segmentation confidence.

8. An electronic device, characterized in that: The electronic device includes a processor coupled to a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable a computer to implement the method according to any one of claims 1 to 4.

Images