A text segmentation and query method and device for RAG system

Abstract

The present application discloses a text segmentation and query method and device for a RAG system, which relates to the field of natural language processing technology. The method includes: receiving a target document; detecting the text type of the currently processed text being read; obtaining the title level corresponding to the title when detecting that the currently processed text is a title; detecting whether the text information in the data item last saved in the storage medium contains data; if there is data in the text information, storing the data item in a resource pool, clearing the text information of the data item and data less than or equal to the title level in the title information, and writing the title to the corresponding position in the title information; if there is no data in the text information, writing the title to the corresponding position in the title information; and storing the currently processed text in the position of the text information in the data item when detecting that the currently processed text is the main text. The present application can improve the completeness and accuracy of the answer.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a text segmentation and query method and device for a RAG system. Background Art

[0002] In recent years, breakthroughs in deep learning have made it possible to build large-scale pre-trained language models. These models are able to capture rich linguistic structure and semantic information through self-supervised learning on large amounts of unlabeled text. However, despite their powerful generative capabilities, these models still have some limitations.

[0003] To overcome the limitations of these models, researchers began exploring how to enable language models to leverage external knowledge bases to enhance their generative capabilities. This requires the model to be able to retrieve the most relevant content from massive amounts of information and incorporate it into the generated text. This demand led to the development of RAG technology.

[0004] In the RAG system, paragraph segmentation of knowledge base document information is generally based on punctuation marks. However, the database obtained by this segmentation method often cannot query all relevant data, resulting in incomplete and inaccurate answers output to users. Summary of the Invention

[0005] The purpose of this application is to provide a text segmentation and query method and device for a RAG system, which can improve the completeness and accuracy of answers.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a text segmentation method for a RAG system, the text segmentation method for a RAG system comprising:

[0008] Receive target document;

[0009] Read each section of the target document in sequence, and perform the following text storage steps on each section of the target document:

[0010] Detecting the text type of the currently read text to be processed, where the text type includes: title or body;

[0011] When it is detected that the current text to be processed is a title, obtaining the title level corresponding to the title;

[0012] Detecting whether there is data in the text information of the last saved data item in the storage medium; the data item indicates the correspondence between title information and text information, the title information in each data item is a title saved in sequence according to title levels, and the text information in each data item is the text corresponding to the lowest-level title in the corresponding title information;

[0013] If there is data in the text information, the data item is stored in a resource pool, the text information of the data item and data less than or equal to the title level in the title information are cleared, the title is written into a corresponding position in the title information, and a detection step is performed;

[0014] If there is no data in the text information, writing the title into the corresponding position in the title information and executing the detection step;

[0015] When it is detected that the current text to be processed is a main text, the current text to be processed is stored at the position of the main text information in the data item, and a detection step is performed;

[0016] The detection step comprises:

[0017] Detect whether the target document has been read completely. If not, re-execute the above text storage step for the next section of text to be processed. If it has been read completely, store the data item in the resource pool.

[0018] In a second aspect, the present application provides a text segmentation device for a RAG system, the text segmentation device for a RAG system comprising:

[0019] A text receiving module, used for receiving a target document;

[0020] The storage module is used to sequentially read each section of the text to be processed in the target document and perform the following text storage steps on each section of the text to be processed:

[0021] Detecting the text type of the currently read text to be processed, where the text type includes: title or body;

[0022] When it is detected that the current text to be processed is a title, obtaining the title level corresponding to the title;

[0023] Detecting whether there is data in the text information of the last saved data item in the storage medium; the data item indicates the correspondence between title information and text information, the title information in each data item is a title saved in sequence according to title levels, and the text information in each data item is the text corresponding to the lowest-level title in the corresponding title information;

[0024] If there is data in the text information, the data item is stored in a resource pool, the text information of the data item and data less than or equal to the title level in the title information are cleared, the title is written into a corresponding position in the title information, and a detection step is performed;

[0025] If there is no data in the text information, writing the title into the corresponding position in the title information and executing the detection step;

[0026] When it is detected that the current text to be processed is a main text, the current text to be processed is stored at the position of the main text information in the data item, and a detection step is performed;

[0027] The detection step comprises:

[0028] Detect whether the target document has been read completely. If not, re-execute the above text storage step for the next section of text to be processed. If it has been read completely, store the data item in the resource pool.

[0029] In a third aspect, the present application provides a text query method for a RAG system, wherein the method applies a resource pool constructed by the text segmentation method for the RAG system as described in the first aspect, and the method comprises:

[0030] Receive inquiries;

[0031] Acquire corresponding data items from the resource pool according to the query question;

[0032] combining the text information in the data item according to the title information and the text information in the data item to obtain an answer corresponding to the query question;

[0033] Output the answer.

[0034] In a fourth aspect, the present application provides a text query device for a RAG system, the text query device for the RAG system comprising:

[0035] A question receiving module, used for receiving query questions;

[0036] A data acquisition module, configured to acquire corresponding data items from the resource pool according to the query question;

[0037] an answer acquisition module, configured to combine the text information in the data item according to the title information and the text information in the data item to obtain an answer corresponding to the query question;

[0038] The answer output module is used to output the answer.

[0039] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0040] The present application provides a text segmentation and query method and device for a RAG system, the method comprising: receiving a target document; sequentially reading each section of text to be processed in the target document, and performing the following text storage steps on each section of text to be processed: detecting the text type of the currently read text to be processed, the text type including: title or text; when detecting that the currently read text to be processed is a title, obtaining the title level corresponding to the title; detecting whether there is data in the text information in the last saved data item in the storage medium; the data item indicates the correspondence between the title information and the text information, the title information in each data item is the title saved in sequence according to the title level, and the text information in each data item is the corresponding title information intermediate level. The text corresponding to the lowest title; if there is data in the text information, the data item is stored in the resource pool, the text information of the data item and the data less than or equal to the title level in the title information are cleared, the title is written into the corresponding position in the title information, and the detection step is executed; if there is no data in the text information, the title is written into the corresponding position in the title information, and the detection step is executed; when it is detected that the current text to be processed is the main text, the current text to be processed is stored in the position of the main text information in the data item, and the detection step is executed; the detection step includes: detecting whether the target document has been read, if not, re-execute the above text storage step for the next paragraph of the text to be processed, if it has been read, store the data item in the resource pool. In the present disclosure, text splitting is performed according to the title granularity, and the data between the two titles represents a paragraph. At the same time, the paragraph title and the parent title are added to increase the retrievable information of the paragraph, and the relevant information is retained to the greatest extent, so that the relevant text can be found more accurately during retrieval, avoiding the loss of text and improving the completeness and accuracy of the answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 The process of the text segmentation method for RAG system according to an exemplary embodiment is shown as follows Figure 1 ;

[0043] Figure 2 is a schematic diagram showing a data structure of a title cache according to an exemplary embodiment;

[0044] Figure 3The process of the text segmentation method for RAG system according to an exemplary embodiment is shown as follows Figure 2 ;

[0045] Figure 4 is a functional module diagram of a text segmentation device for a RAG system according to an exemplary embodiment;

[0046] Figure 5 is a flowchart of a text query method of a RAG system according to an exemplary embodiment;

[0047] Figure 6 is a functional module diagram of a text query device of a RAG system according to an exemplary embodiment;

[0048] Figure 7 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0051] In recent years, breakthroughs in deep learning have made it possible to build large-scale pre-trained language models, such as BERT and GPT. These models capture rich linguistic structure and semantic information through self-supervised learning on large amounts of unlabeled text. However, despite their powerful generative capabilities, these models still have some limitations, such as:

[0052] Knowledge limitations: The knowledge of pre-trained models is usually limited to the time range of their training data and cannot be updated with new knowledge in real time.

[0053] Factual bias: Due to bias or deficiencies in the training data, the model may generate inaccurate or false factual information.

[0054] Fabrication tendency: The model tends to generate fabricated content, especially in the absence of specific context.

[0055] To overcome these limitations, researchers began exploring how to enable language models to leverage external knowledge bases to enhance their generative capabilities. This requires the model to be able to retrieve the most relevant content from massive amounts of information and incorporate it into the generated text. This demand has led to the development of RAG technology.

[0056] Currently, the RAG system uses the following text segmentation methods to generate vector databases:

[0057] The first method is to directly use the five text segmentation methods provided by Langchain. The general methods are as follows:

[0058] 1) Convert the document into text using different file processing programs such as Word reader, Orc reader, and Markdown reader;

[0059] 2) Input the text into the text segmentation tool;

[0060] 3) The text segmentation tool divides the text into text blocks and stores them in the vector database.

[0061] However, this segmentation method has the following disadvantages:

[0062] 1) Improper segmentation: If fixed-length segmentation is used, two closely related texts will be split into two paragraphs. It is possible that one paragraph cannot be retrieved during the query, resulting in problems with the answer.

[0063] 2) Slow efficiency: If the model is used for segmentation, it will seriously affect the efficiency of data writing and querying.

[0064] 3) Segmentation accuracy: Models are usually trained in English. Chinese semantics and grammar differ from English, which ultimately leads to inaccurate segmented data.

[0065] The second method is to use punctuation marks and the features of the upper and lower texts for segmentation. The general method is as follows:

[0066] 1. Obtain the first text T1 of the paragraph to be segmented, and clean the first text T1 of the paragraph to be segmented to obtain the second text T2 of the paragraph to be segmented;

[0067] 2. Segment the second text T2 of the paragraph to be segmented according to the punctuation marks of the second text T2 of the paragraph to be segmented, and generate a sentence list; the sentence list includes: a plurality of sentences formed by the segmentation of the second text T2 of the paragraph to be segmented;

[0068] 3. Traverse each sentence in the sentence list, process each sentence in the sentence list, and construct a paragraph list; the paragraph list includes: a plurality of new paragraphs obtained after processing each sentence in the sentence list;

[0069] 4. Traverse the paragraph list, clean each new paragraph in the paragraph list, and generate a paragraph pool.

[0070] However, this technical solution has the following disadvantages:

[0071] 1) Documents use irregular punctuation: When a document begins with a bullet point and the previous paragraph is separated by a period, this may result in illogical segmentation. This may cause semantically related paragraphs to be split into two, leading to query problems.

[0072] 2) Semantic segmentation: Sentences are segmented solely by punctuation, so all relevant data cannot be retrieved during query. For example, the following text:

[0073] A Hospital's Regulations on Employee Working Hours

[0074] Article 1: These Regulations are formulated in accordance with the relevant provisions of Law B in order to reasonably arrange the working hours and rest time of employees, safeguard the employees' right to rest, mobilize the enthusiasm of employees and promote the development of the cause.

[0075] Article 2: These regulations apply to employees of state organs, social groups, enterprises, institutions and other organizations within the territory of Country C.

[0076] Article 3: Employees work 8 hours a day and 40 hours a week.

[0077] According to the second method, the first, second and third items will each generate a paragraph. When searching, if the user enters the question "What are the working hours regulations", the first and third items can be searched, but the second item cannot be searched. This is because the second item has little relevance to the query question.

[0078] In order to solve the above technical problems, the present disclosure proposes a text segmentation and query method and device for a RAG system.

[0079] Figure 1 is a flowchart of a text segmentation method for a RAG system according to an exemplary embodiment. Figure 1 As shown, the method includes the following steps S101-S108:

[0080] In step S101 , a target document is received.

[0081] Each section of the text to be processed in the target document is read in sequence, and the following text storage steps S102-S107 are performed on each section of the text to be processed:

[0082] In step S102, the text type of the currently read text to be processed is detected, and the text type includes: title or text.

[0083] After receiving the target document, the target document may be parsed by a corresponding document parser to obtain the format of the target document.

[0084] Document formats may include: word processing document format, spreadsheet document format, presentation document format, and portable document format.

[0085] Among them, word processing document formats may include: DOC / DOCX, TXT and RTF (Rich Text Format);

[0086] Spreadsheet document formats can include: XLS / XLSX and CSV (Comma-Separated Values);

[0087] Presentation document formats include: PPT / PPTX and ODP (Open Document Presentation);

[0088] Portable document formats may include: PDF (Portable Document Format).

[0089] After receiving the target document and identifying the document format of the target document, each section of the text to be processed in the target document is read in turn. After reading the current text to be processed, the text type of the current text to be processed needs to be identified first, that is, it needs to be identified whether the current text to be processed is a title or a body text. There will be different processing solutions based on the title and the body text.

[0090] In step S103, when it is detected that the current text to be processed is a title, the title level corresponding to the title is obtained.

[0091] In a document, heading levels are used to organize the content hierarchically, making it easier for readers to quickly understand the document's main content and key points. Different document formats and editing software use different methods for setting heading levels, but the basic principles are similar.

[0092] First-level headings (usually correspond to the highest level of the document outline).

[0093] Style Characteristics: These are typically large and bold, making them stand out on the page. They often directly reflect the document's theme and summarize the core content. For example, in an academic paper, a first-level title might be "On the Application of Artificial Intelligence in the Medical Field," which clearly defines the scope of the paper.

[0094] Layout Position: Typically located at the beginning of a document or at the start of a new chapter. There may be a large gap between it and the main text to enhance visual distinction. In some document formats (such as Markdown), first-level headings are often marked with a "#" symbol, such as "#Applications of Artificial Intelligence in the Medical Field."

[0095] Purpose: Helps readers quickly grasp the key points of a document and determine whether it is relevant to their needs. It also forms an important foundation for building a document table of contents. Many word processing software can automatically generate a document table of contents based on the first-level headings.

[0096] Secondary title.

[0097] Style Features: The font size is between the first-level heading and the main text, and may also be bold. It is a subdivision of the first-level heading, used to explain the main sub-topics within the topic. For example, in the academic paper mentioned earlier, the second-level heading might be "Application of Artificial Intelligence in Disease Diagnosis," which is a key sub-topic within the larger topic of "Application of Artificial Intelligence in the Medical Field."

[0098] Layout Position: After the first-level heading, the content of the first-level heading is usually followed, and there is a certain distance between the first-level heading and the main text to reflect the hierarchy. In markdown, "##" is often used to mark the second-level heading, such as "## Application of Artificial Intelligence in Disease Diagnosis".

[0099] Purpose: Further refines the document content, allowing readers to more clearly understand the document structure. For long, complex documents, secondary headings can quickly guide readers to the sub-content they are interested in.

[0100] Level 3 headings and below.

[0101] Style Features: Font size and style will decrease in descending order based on the overall style and level of the document. Some software may maintain bold fonts, but the degree of bolding will gradually decrease. These headings provide a more detailed division of the content of the upper-level heading. For example, under the second-level heading "Application of Artificial Intelligence in Disease Diagnosis," the third-level heading might be "Disease Imaging Diagnosis Based on Deep Learning," which more specifically describes an application method in the sub-field of disease diagnosis.

[0102] Layout Position: Nested within the content of the parent heading, maintaining appropriate spacing between the parent heading and the body of the text. In markdown, third-level headings are marked with "###", such as "### Disease Imaging Diagnosis Based on Deep Learning".

[0103] Benefits: As heading levels are broken down, complex document content can be broken down into smaller, easier-to-understand sections, making it easier for both authors and readers to write and read. For content-rich documents like technical documentation and long reports, lower-level headings help build a detailed knowledge structure and make the document content more organized.

[0104] Here are some ways to identify heading levels in your document:

[0105] 1. Identify through format features.

[0106] 1. Font and font size.

[0107] Generally speaking, the higher the heading level, the larger the font size. First-level headings typically use the largest font size. For example, in many documents, first-level headings might be in size 2 or 1, while second-level headings might be in size 3, and third-level headings in size 4. Furthermore, heading fonts are often eye-catching, perhaps bold or black. For example, in a company report, the first-level heading "Annual Work Summary" is bold and large, while the second-level heading "Sales Performance Analysis" is slightly smaller and also bold. By comparing the font size and boldness, you can distinguish between heading levels.

[0108] 2. Color and style.

[0109] Some documents use color to distinguish heading levels. For example, a first-level heading might be dark, like black, while a second-level heading might be a lighter color, like gray. Headings might also have special styling, such as underlining, shading, or using different fonts (e.g., Songti for a first-level heading and Kaiti for a second-level heading) to differentiate them. However, this is relatively uncommon, as excessive color and style variations can make a document look cluttered.

[0110] 3. Paragraph format.

[0111] The paragraph spacing between the title and the text can be used as a basis for identification. The first-level title usually has a larger spacing from the text, followed by the second-level title, and the third-level title has an even smaller spacing. Moreover, the paragraph indentation of the title itself may also be different. For example, the first-level title may be centered, the second-level title may be left-aligned and have a certain indentation, and the third-level title may have more indentation. Taking an academic paper as an example, the first-level title "Introduction" is centered on the page, with more blank lines between it and the text below; the second-level title "Research Background" is left-aligned and has a slightly smaller spacing from the text below; the third-level title "Current Status of Domestic and Foreign Research" is indented and has an even smaller spacing from the text.

[0112] 2. Identify through document structure and content logic.

[0113] 1. Content hierarchy.

[0114] Observing the scope of content covered by a title can help identify the heading level. A first-level heading typically provides a high-level summary of the entire document's topic and has the broadest scope. A second-level heading is a breakdown of the topic covered by the first-level heading, narrowing its scope. For example, in a product manual, the first-level heading is "Product Overview," while the second-level headings "Product Features" and "Product Technical Specifications" are specific sections broken down from the larger topic of the product overview. This inclusion relationship of content can be used to determine the heading level.

[0115] 2. Numbering system.

[0116] Some documents use numbering to indicate heading levels. For example, a numbering system like "1., 1.1, 1.1.1" might be used. "1." typically represents a first-level heading, "1.1" represents a second-level heading, and "1.1.1" represents a third-level heading. This numbering system clearly illustrates the hierarchical relationship between headings. For example, in a project plan, "1. Project Objectives" is a first-level heading, "1.1 Short-term Objectives" is a second-level heading, and "1.1.1 Phase One Objectives" is a third-level heading.

[0117] 3. Grammatical and vocabulary characteristics.

[0118] First-level titles are often abstract and general, using general terms. Second-level titles are more specific, and third-level titles are even more detailed. For example, a first-level title might be "The Impact of Education Reform," a second-level title might be "The Impact on Teaching Methods," and a third-level title might be "The Specific Application of New Teaching Methods in the Classroom." The wording used in these titles indicates a progression from broad to specific, helping to identify the title level.

[0119] 3. Use software functions to identify.

[0120] 1. Outline view (in word processing software).

[0121] Many word processors offer an outline view. In outline view, the software automatically displays the headings in a hierarchical manner, based on the document's heading levels. Level 1 headings appear at the highest level, level 2 headings appear at the next level, and so on. Outline view allows users to clearly see the hierarchy of headings and easily adjust heading levels.

[0122] 2. Style function (in word processing software).

[0123] The style feature in the software can help identify heading levels. When writing a document, the author may use the software's preset styles or custom styles to set the headings. By looking at the style name (such as "Heading 1", "Heading 2", etc.) or style attributes (such as font, font size, etc.), you can determine the heading level. For example, in word processing software, the "Heading 1" style is usually used for first-level headings, and the "Heading 2" style is used for second-level headings. By checking which style is applied to the heading, you can accurately identify the heading level.

[0124] The present disclosure may identify the title level through any of the above methods.

[0125] In step S104, it is detected whether there is data in the text information of the data item saved last time in the storage medium; the data item indicates the correspondence between the title information and the text information, the title information in each data item is the title saved in sequence according to the title level, and the text information in each data item is the text corresponding to the title with the lowest level in the corresponding title information.

[0126] The format of the data items will be pre-set in the storage medium, as shown in Table 1:

[0127]

[0128] In Table 1, 4 levels of titles are preset, that is, when the recognized title level is level 1, it is filled in the position corresponding to the level 1 title, and when the recognized title level is level 2, it is filled in the position corresponding to the level 2 title.

[0129] In step S105, if there is data in the text information, the data item is stored in the resource pool, the text information of the data item and the data less than or equal to the title level in the title information are cleared, the title is written to the corresponding position in the title information, and the detection step of step S108 is executed, that is, whether the target document has been read. If not, the above text storage step is re-executed for the next paragraph of text to be processed. If it has been read, the data item is stored in the resource pool.

[0130] After identifying the level of the title, it is also necessary to detect whether there is data in the text information of the last data item in the storage medium. If there is data in the text information of the last saved data item, it indicates that the last identification was the text corresponding to the previous title. Moreover, since the identification this time is the title, it indicates that the text corresponding to the previous title has been identified and needs to be stored in the resource pool for subsequent retrieval. After saving the last data item to the resource pool, clear the text information of the data item and the data in the title information that is less than or equal to the title level, and write the title to the corresponding position in the title information.

[0131] Take the following target document as an example:

[0132] 1.AAA.

[0133] 1.1BB.

[0134] Bbb.

[0135] 1.2CC.

[0136] Ccc.

[0137] Among them, 1.AAA is a level 1 title, 1.1BB is a level 2 title, Bbb is the text corresponding to level 2 title BB, 1.2CC is also a level 2 title, Ccc is the text corresponding to level 2 title CC, 1.AAA is the first paragraph of text to be processed, 1.1BB is the second paragraph of text to be processed, Bbb is the third paragraph of text to be processed, 1.2CC is the fourth paragraph of text to be processed, and Ccc is the fifth paragraph of text to be processed.

[0138] The data items in the storage medium are shown in Table 2:

[0139]

[0140] Assume that the fourth segment of text to be processed is currently being recognized: 1.2CC.

[0141] Identify 1.2CC as a title, and the title level is 2. Detect that the text information in the last saved data item (Table 2) in the storage medium contains data. Then save Table 2 to the resource pool, clear the text information of the data item and the data less than or equal to the title level in the title information, and obtain Table 3:

[0142]

[0143] Write CC into the position corresponding to the level 2 title to obtain Table 4:

[0144]

[0145] In one embodiment, storing a data item in a resource pool includes the following sub-steps A1-A5:

[0146] A1. Check whether the number of characters in the text information in the data item is greater than a preset number of characters.

[0147] A2. If the number of characters is greater than a preset number, the characters in the text message are divided into multiple text blocks according to the preset number.

[0148] A3. Divide the data item into a corresponding number of sub-data items according to the number of text blocks. Each sub-data item includes: title information and current text block information. The current text block information includes: current text block, total number of text blocks and order of current text block.

[0149] A4. Store the sub-data item in the resource pool.

[0150] A5. If the number of characters is not greater than the preset number, the data item is stored in the resource pool.

[0151] The sub-data items are shown in Table 5:

[0152]

[0153] Table 6 is used as an example for explanation:

[0154]

[0155] When Table 6 needs to be saved to the resource pool, the main text 111222333444555 can be divided into multiple text blocks. Assuming the preset number is 3, the main text 111222333444555 is divided into 5 text blocks in order, namely: 111, 222, 333, 444 and 555, among which 111 is ranked 1, 222 is ranked 2, 333 is ranked 3, 444 is ranked 4, and 555 is ranked 5, obtaining 5 sub-data items as shown in Table 7, and then these 5 sub-data items are stored in the resource pool respectively.

[0156]

[0157] In one embodiment, the current text block information may further include: the last preset number of characters of the previous text block.

[0158] The sub-data items may also be as shown in Table 8, where the last two characters of the previous text block are added:

[0159]

[0160] Continuing to use Table 6 as an example, when Table 6 needs to be saved to the resource pool, the sub-data items obtained at this time are shown in Table 9:

[0161]

[0162] Furthermore, the data items may be vectorized and then stored in the resource pool, and only the title information and the current text block may be vectorized.

[0163] To ensure that the vector can express semantics as much as possible, the vectorized text needs to be kept within a certain range. Therefore, if the paragraph is too large, it is necessary to split it, and then the order of the text blocks and the title information need to be retained after the split.

[0164] In step S106, if there is no data in the text information, the title is written to the corresponding position in the title information, and the detection step of step SS108 is executed, that is, whether the target document has been read. If not, the above text storage step is re-executed for the next section of text to be processed. If it has been read, the data item is stored in the resource pool.

[0165] Specifically, writing the title into the corresponding position in the title information includes the following sub-steps B1-B3:

[0166] B1. Check whether there is data in the title information;

[0167] B2. If there is no data in the header information, write the header to the corresponding position in the header information;

[0168] B3. If there is data in the title information, clear the data in the title information of the data item that is less than or equal to the title level, and write the title to the corresponding position in the title information.

[0169] If there is no data in the text information, it indicates that the last recognition was for a title. Then the title is written to the corresponding position in the title information. If data has been written to the position with the same title level as the title in the data item, the title at the same title level and the lower-level position must be deleted, and the title is written to the corresponding position in the title information.

[0170] Take the following target document as an example:

[0171] 1.AAA.

[0172] 2.DDD.

[0173] Among them, 1.AAA is a level 1 title with no text, and 2.DDD is a level 1 title.

[0174] The data items are shown in Table 10:

[0175]

[0176] Assume that the second segment of text to be processed is currently recognized: 2.DDD.

[0177] Identify 2.DDD as a title, and the title level is level 1. Detect that there is no data in the text information of the last saved data item (Table 5) in the storage medium. Then clear the data AAA corresponding to the level 1 title, and fill DDD into the position corresponding to the level 1 title, and obtain Table 11:

[0178]

[0179] In step S107, when it is detected that the current text to be processed is the main text, the current text to be processed is stored in the position of the main text information in the data item, and the detection step of step S108 is executed, that is, whether the target document has been read. If it has not been read, the above text storage step is re-executed for the next piece of text to be processed. If it has been read, the data item is stored in the resource pool.

[0180] If it is detected that the current text to be processed is the main text, it means that this is the content after the last recognized content. If the last recognized content was the title, it is the main text corresponding to the title. If the last recognized content was the main text, this time it is the next paragraph of the main text. The current text to be processed is stored in the position of the main text information in the data item.

[0181] Take the following target document as an example:

[0182] 1.AAA.

[0183] Aaa.

[0184] Ddd.

[0185] Among them, 1.AAA is the level 1 title, Aaa is the first paragraph of text corresponding to the level 1 title, and Ddd is the second paragraph of text corresponding to the level 1 title.

[0186] The data items are shown in Table 12:

[0187]

[0188] Assume that the third paragraph of text to be processed is currently recognized: Ddd.

[0189] Identify Ddd as text, detect that the text information in the last saved data item (Table 2) in the storage medium has data, then write Ddd to the position corresponding to the text, and obtain Table 13:

[0190]

[0191] In step S108, it is detected whether the target document has been read. If not, the above text storage step is re-executed for the next section of text to be processed. If it has been read, the data item is stored in the resource pool.

[0192] In the present disclosure, the text is split according to the title granularity, and the data between two titles represents a paragraph. At the same time, the paragraph title and the parent title are added to increase the searchable information of the paragraph and retain relevant information to the greatest extent possible, so that relevant text can be found more accurately during retrieval, avoiding the loss of text.

[0193] Since the text is split based on the title as the granularity, the semantic loss between texts caused by fixed length is avoided, and the present disclosure can accurately recognize Chinese, thereby avoiding the disadvantage of some models having low support for Chinese.

[0194] In one implementation, the storage medium in the present disclosure is a cache.

[0195] Cache is the ability of a computer to temporarily store frequently accessed data or calculation results to reduce latency and improve performance.

[0196] The cache of header information is as follows Figure 2 The data structure shown in the figure is implemented using an array in a computer data structure. Figure 2 The level 1 title is in the first position of the array, the level 2 title is in the second position of the array, and so on. When clearing titles after the nth level, add 1 to the index and then set the current value to empty; when querying a title of a specified level, you can directly get the corresponding title according to the level.

[0197] The cache of the text information can also add corresponding end characters at the end of the text when it is actually used.

[0198] When the document is first recognized, the cache of the title information and the cache of the body information may also be initialized.

[0199] Figure 3 is a flowchart of a text segmentation method for a RAG system according to an exemplary embodiment. Figure 3 As shown, the basic steps of the entire technical solution for text segmentation are as follows:

[0200] 1> Document parsing: parse the document through the corresponding text parser.

[0201] 2> Initialize the title cache and initialize the body cache.

[0202] The title information cache mainly stores the actual title content corresponding to different levels of titles, such as the first-level title, the second-level title, etc. When the title changes, each level of title will dynamically change. The body cache stores entire paragraphs with the same semantics. The format of the title cache is <title level: title content>.

[0203] The title cache here represents the title information in the above embodiment, and the text cache here represents the text information in the above embodiment.

[0204] 3> Read a piece of text: determine whether the text is the main text or the title.

[0205] 3.1> If it is a title.

[0206] 3.1.1> Check whether there is content in the body cache.

[0207] If the body cache has content, go to 3.1.1.1. If the body cache has no content, go to 3.1.2.

[0208] 3.1.1.1> Split the text in the body buffer into text blocks according to the specified length, [last 20 characters of the previous text block: current text block: total number of text blocks: sorting of the current text block].

[0209] Split the text into blocks of the specified size. Each block is defined according to the following rules:

[0210] First-level title -> Second-level title -> Third-level title ->… -> N-level title: The last 20 characters of the previous text block: Current text block: The total number of text blocks: The sorting of the current text block.

[0211] 3.1.1.2> Save the title information and text block together and store them in the vector database in the form of [title information: text information].

[0212] The title information here is: first-level title -> second-level title -> third-level title ->… -> n-level title; the text information here is: the last 20 characters of the previous text block: the current text block: the total number of text blocks: the order of the current text block.

[0213] The vectorized part is the first-level title -> second-level title -> third-level title ->… -> n-level title: the last 20 characters of the previous text block: the current text block.

[0214] The total number of text blocks and the order of the current text block are not vectorized.

[0215] 3.1.2> If it is detected that there is no content in the text cache, then the text cache will be directly cleared, the titles that are less than or equal to the level will be cleared, and the corresponding title information will be updated.

[0216] 3.2>If it is text: add the current text to the text cache.

[0217] 4> Determine whether there is any text.

[0218] 4.1> If there is still text, jump back to 3.

[0219] 4.2> If there is no text, jump to 5.

[0220] 5> Determine whether there is content in the text cache.

[0221] 5.1>If there is content.

[0222] 5.1.1> Split the text in the main text buffer into text blocks according to the specified length, [the last 20 characters of the previous text block: the current text block: the total number of text blocks: the order of the current text block].

[0223] 5.1.2>After the definition is completed, save the title information and text block together and store them in the vector database in the form of [title information: text information].

[0224] The title information here is: first-level title -> second-level title -> third-level title ->… -> n-level title; the text information here is: the last 20 characters of the previous text block: the current text block: the total number of text blocks: the order of the current text block.

[0225] The vectorized part is the first-level title -> second-level title -> third-level title ->… -> n-level title: the last 20 characters of the previous text block: the current text block.

[0226] The total number of text blocks and the order of the current text block are not vectorized.

[0227] 5.1.3> Jump to 6.

[0228] 5.2> If there is no content, jump to 6.

[0229] 6>End.

[0230] By adopting the method disclosed in the present invention, a complete and accurate knowledge base can be established, providing a good foundation for future enterprise-level knowledge retrieval.

[0231] Based on the same inventive concept, an embodiment of the present application further provides a text segmentation device for a RAG system for implementing the above-mentioned text segmentation method for a RAG system. The implementation solution provided by the device is similar to the implementation solution described in the above-mentioned method. Therefore, the specific limitations in one or more embodiments of the text segmentation device for a RAG system provided below can be found in the above-mentioned limitations on the text segmentation method for a RAG system, and will not be repeated here.

[0232] In an exemplary embodiment, Figure 4 As shown, a text segmentation device for a RAG system is provided, comprising:

[0233] A text receiving module 11 is used to receive a target document;

[0234] The storage module 12 is used to sequentially read each section of the text to be processed in the target document and perform the following text storage steps on each section of the text to be processed:

[0235] Detect the text type of the currently read text to be processed, the text type includes: title or body;

[0236] When it is detected that the current text to be processed is a title, obtain the title level corresponding to the title;

[0237] Detecting whether there is data in the text information of the last saved data item in the storage medium; the data item indicates the correspondence between the title information and the text information, the title information in each data item is the title saved in sequence according to the title level, and the text information in each data item is the text corresponding to the title with the lowest level in the corresponding title information;

[0238] If there is data in the text information, the data item is stored in the resource pool, the text information of the data item and the data less than or equal to the title level in the title information are cleared, and the title is written into the corresponding position in the title information;

[0239] If there is no data in the text information, writing the title into the corresponding position in the title information;

[0240] When it is detected that the current text to be processed is a main text, storing the current text to be processed at the position of the main text information in the data item;

[0241] Detect whether the target document has been read completely. If not, re-execute the above text storage step for the next section of text to be processed. If it has been read completely, store the data item in the resource pool.

[0242] In one embodiment, in terms of writing the title into a corresponding position in the title information, the storage module 12 is specifically configured to:

[0243] detecting whether there is data in the title information;

[0244] If there is no data in the title information, writing the title into a corresponding position in the title information;

[0245] If there is data in the title information, clear the data in the title information of the data item that is less than or equal to the title level, and write the title into the corresponding position in the title information.

[0246] In one embodiment, in the aspect of storing the data item in the resource pool, the storage module 12 is specifically configured to:

[0247] Detecting whether the number of characters in the text information in the data item is greater than a preset number of characters;

[0248] If the number of characters is greater than the preset number, dividing the characters in the text message into multiple text blocks according to the preset number;

[0249] Dividing the data item into a corresponding number of sub-data items according to the number of the text blocks, each of the sub-data items includes: the title information and current text block information, the current text block information including: the current text block, the total number of the text blocks, and the order of the current text block;

[0250] Storing the sub-data item in the resource pool;

[0251] If it is not greater than the preset number of characters, the data item is stored in the resource pool.

[0252] In one embodiment, the current text block information further includes: the last preset number of characters of the previous text block.

[0253] In one embodiment, in the aspect of storing the data item in the resource pool, the storage module 12 is specifically configured to:

[0254] The data items are vectorized and stored in the resource pool.

[0255] In one embodiment, the storage medium is a cache.

[0256] Figure 5 FIG. 1 is a flowchart of a text query method of a RAG system according to an exemplary embodiment. Figure 5 As shown, the method applies a resource pool constructed by the text segmentation method for the RAG system in any of the above embodiments, and the method includes the following sub-steps S201-S204:

[0257] S201: Receive query questions.

[0258] S202: Obtain corresponding data items from the resource pool according to the query question.

[0259] After receiving the user's query question, based on the question, the corresponding data item is obtained from the resource pool. After obtaining the data item.

[0260] S203: Combine the text information in the data item according to the title information and text information in the data item to obtain an answer corresponding to the query question.

[0261] After the corresponding data items are found in the resource pool, the data items are spliced ​​together to obtain the answer to the query question.

[0262] S204. Output the answer.

[0263] In the present disclosure, the RAG system includes a RAG module and an LLM module. The LLM module receives a query question and sends the question to the RAG module. The RAG module obtains the corresponding data item from the resource pool according to the query question, obtains the answer corresponding to the query question according to the data item, and then sends the answer to the LLM module. The LLM module outputs the answer.

[0264] In one embodiment, combining the text information in the data item according to the title information and the text information in the data item to obtain the answer corresponding to the query question includes the following sub-steps C1-C5:

[0265] C1. Grouping the acquired data items according to title information and text information.

[0266] C2. Detecting whether the text information is missing based on the total number of text blocks in the text information and the order of the current text block.

[0267] C3. If yes, search the resource pool again for the target data item corresponding to the missing information according to the title information, the total number of the text blocks and the order of the current text block.

[0268] The queried data items are grouped according to the title information and text information, and a judgment is made based on the total number of text blocks of each text and the order of each text block to find out whether there are any missing text blocks. If there are missing text blocks, a search is performed again based on the last preset number of characters of the previous text block, the title information and the order of the text blocks to find the data items corresponding to the missing information.

[0269] C4. Concatenate the information in the data item and the target data item to obtain an answer corresponding to the query question.

[0270] C5. If not, combine the text information in the data item according to the title information and text information in the data item to obtain an answer corresponding to the query question.

[0271] In one embodiment, obtaining corresponding data items from the resource pool according to the query question includes the following sub-steps D1-D2:

[0272] D1. Vectorize the query question to obtain a vectorized question.

[0273] D2. Obtain corresponding data items from the resource pool according to the vectorized problem.

[0274] After receiving the query question, the query question can be vectorized, and the resource pool can be queried according to the vectorized vector of the question.

[0275] When merging, you need to combine the information retained during the split to ensure that the paragraphs are in order. This prevents the LLM from being given a disordered state after the search, which could lead to misunderstandings and incorrect steps. Also, when combining, you need to complete any missing parts to ensure that the LLM doesn't lose any information when answering.

[0276] Based on the same inventive concept, an embodiment of the present application further provides a text query device for a RAG system for implementing the text query method for the RAG system involved above. The implementation solution provided by the device is similar to the implementation solution described in the above method. Therefore, the specific limitations in the embodiments of one or more text query devices for RAG systems provided below can be found in the above-mentioned limitations on the text query method for the RAG system, and will not be repeated here.

[0277] In an exemplary embodiment, Figure 6 As shown, a text query device of a RAG system is provided, comprising:

[0278] The question receiving module 21 is used to receive query questions.

[0279] The data acquisition module 22 is configured to acquire corresponding data items from the resource pool according to the query question.

[0280] The answer acquisition module 23 is configured to combine the text information in the data item according to the title information and the text information in the data item to obtain the answer corresponding to the query question.

[0281] The answer output module 24 is configured to output the answer.

[0282] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 7As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a text query method of a RAG system is implemented.

[0283] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0284] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0285] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0286] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0287] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0288] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0289] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0290] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0291] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A text segmentation method for a RAG system, characterized in that: The text segmentation method for the RAG system includes: Receive target document; Read each section of the text to be processed in the target document in sequence, and perform the following text storage steps on each section of the text to be processed: Detecting the text type of the currently read text to be processed, where the text type includes: title or body; When it is detected that the current text to be processed is a title, obtaining the title level corresponding to the title; Detecting whether there is data in the text information of the last saved data item in the storage medium; the data item indicates the correspondence between title information and text information, the title information in each data item is a title saved in sequence according to title levels, and the text information in each data item is the text corresponding to the lowest-level title in the corresponding title information; If there is data in the text information, the data item is stored in a resource pool, the text information of the data item and data less than or equal to the title level in the title information are cleared, the title is written into a corresponding position in the title information, and a detection step is performed; If there is no data in the text information, writing the title into the corresponding position in the title information and executing the detection step; When it is detected that the current text to be processed is a main text, the current text to be processed is stored at the position of the main text information in the data item, and the detecting step is performed; The detection step comprises: Detecting whether the target document has been read completely; if not, re-performing the above text storage step for the next section of text to be processed; if the reading has been completed, storing the data item in the resource pool; Storing the data item in a resource pool includes: Detecting whether the number of characters in the text information in the data item is greater than a preset number of characters; If the number of characters is greater than the preset number, dividing the characters in the text message into multiple text blocks according to the preset number; Dividing the data item into a corresponding number of sub-data items according to the number of the text blocks, each sub-data item includes: the title information and the current text block information, the current text block information including: the current text block, the total number of the text blocks, the ranking of the current text block, and the last preset number of characters of the previous text block; when querying, detecting whether the queried text information is missing based on the total number of text blocks and the ranking of the current text block; if so, searching for the missing information based on the title information, the total number of text blocks, and the ranking of the current text block; Storing the sub-data item in the resource pool; If it is not greater than the preset number of characters, the data item is stored in the resource pool.

2. The text segmentation method for RAG system according to claim 1, characterized in that: Writing the title into a corresponding position in the title information includes: detecting whether there is data in the title information; If there is no data in the title information, writing the title into a corresponding position in the title information; If there is data in the title information, clear the data in the title information that is less than or equal to the title level, and write the title into the corresponding position in the title information.

3. The text segmentation method for RAG system according to claim 1, characterized in that: Storing the data item in a resource pool includes: The data items are vectorized and stored in the resource pool.

4. The text segmentation method for RAG system according to claim 1, characterized in that: The storage medium is a cache.

5. A text segmentation device for a RAG system, characterized in that: The text segmentation device for the RAG system includes: A text receiving module, used for receiving a target document; The storage module is used to sequentially read each section of the text to be processed in the target document and perform the following text storage steps on each section of the text to be processed: Detecting the text type of the currently read text to be processed, where the text type includes: title or body; When it is detected that the current text to be processed is a title, obtaining the title level corresponding to the title; Detecting whether there is data in the text information of the last saved data item in the storage medium; the data item indicates the correspondence between title information and text information, the title information in each data item is a title saved in sequence according to title levels, and the text information in each data item is the text corresponding to the lowest-level title in the corresponding title information; If there is data in the text information, the data item is stored in a resource pool, the text information of the data item and data less than or equal to the title level in the title information are cleared, the title is written into a corresponding position in the title information, and a detection step is performed; If there is no data in the text information, writing the title into the corresponding position in the title information and executing the detection step; When it is detected that the current text to be processed is a main text, the current text to be processed is stored at the position of the main text information in the data item, and the detecting step is performed; The detection step comprises: Detecting whether the target document has been read completely; if not, re-performing the above text storage step for the next section of text to be processed; if the reading has been completed, storing the data item in the resource pool; In the aspect of storing the data item in a resource pool, the storage module is specifically configured to: Detecting whether the number of characters in the text information in the data item is greater than a preset number of characters; If the number of characters is greater than the preset number, dividing the characters in the text message into multiple text blocks according to the preset number; Dividing the data item into a corresponding number of sub-data items according to the number of the text blocks, each of the sub-data items includes: the title information and current text block information, the current text block information including: the current text block, the total number of the text blocks, the order of the current text block, and the last preset number of characters of the previous text block; Storing the sub-data item in the resource pool; If it is not greater than the preset number of characters, the data item is stored in the resource pool.

6. A text query method for a RAG system, characterized in that: The method applies a resource pool constructed by the text segmentation method for the RAG system according to any one of claims 1 to 4, and the method comprises: Receive inquiries; Acquire corresponding data items from the resource pool according to the query question; combining the text information in the data item according to the title information and the text information in the data item to obtain an answer corresponding to the query question; outputting the answer; Combining the text information in the data item according to the title information and the text information in the data item to obtain an answer corresponding to the query question includes: Grouping the acquired data items according to title information and text information; detecting whether the text information is missing according to the total number of text blocks in the text information and the order of the current text block; If yes, searching the resource pool again for the target data item corresponding to the missing information according to the title information, the total number of the text blocks and the order of the current text block; Joining the information in the data item and the target data item to obtain an answer corresponding to the query question; If not, the text information in the data item is combined according to the title information and text information in the data item to obtain the answer corresponding to the query question.

7. A text query device for a RAG system, characterized in that: The text query device of the RAG system includes: A question receiving module, used for receiving query questions; A data acquisition module, configured to acquire corresponding data items from a resource pool according to the query question; an answer acquisition module, configured to combine the text information in the data item according to the title information and the text information in the data item to obtain an answer corresponding to the query question; An answer output module, used for outputting the answer; In the aspect of combining the text information in the data item according to the title information and the text information in the data item to obtain the answer corresponding to the query question, the answer acquisition module is specifically configured to: Grouping the acquired data items according to title information and text information; detecting whether the text information is missing according to the total number of text blocks in the text information and the order of the current text block; If yes, searching the resource pool again for the target data item corresponding to the missing information according to the title information, the total number of the text blocks and the order of the current text block; Joining the information in the data item and the target data item to obtain an answer corresponding to the query question; If not, the text information in the data item is combined according to the title information and text information in the data item to obtain the answer corresponding to the query question.

Images