Text segmentation and query method and device for RAG system

Abstract

The invention discloses a text segmentation and query method and device for an RAG system, and relates to the technical field of natural language process.The method comprises the steps that a target document is received; detecting the text type of the read current to-be-processed text; when it is detected that the current to-be-processed text is a title, obtaining a title level corresponding to the title; detecting whether the text information in the data item stored last time in the storage medium has data or not; if data exist in the text information, data items are stored in a resource pool, data, smaller than or equal to the title level, in the text information and title information of the data items are emptied, and titles are written into corresponding positions in the title information; if no data exist in the text information, the title is written into the corresponding position in the title information; and when it is detected that the current to-be-processed text is the text, storing the current to-be-processed text at the position of the text information in the data item. The answer integrity and accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a text segmentation and query method and device for a RAG system. Background Art

[0002] In recent years, breakthroughs in deep learning have made it possible to build large-scale pre-trained language models. These models can capture rich language structure and semantic information by performing self-supervised learning on a large amount of unlabeled text. However, despite their powerful generative capabilities, these models still have some limitations.

[0003] In order to overcome the limitations of the above models, researchers began to explore how to enable language models to use external knowledge bases to enhance their generation capabilities. This requires the model to be able to retrieve the most relevant content from massive information and incorporate it into the generated text. This demand has led to the development of RAG technology.

[0004] In the RAG system, paragraph segmentation of knowledge base document information is generally performed based on punctuation marks. However, the database obtained by this segmentation method often cannot query all relevant data, resulting in incomplete and inaccurate answers output to users. Summary of the invention

[0005] The purpose of this application is to provide a text segmentation and query method and device for a RAG system, which can improve the completeness and accuracy of the answer.

[0006] To achieve the above objectives, this application provides the following solutions: In a first aspect, the present application provides a text segmentation method for a RAG system, and the text segmentation method for a RAG system comprises: receiving the target document; Read each section of the text to be processed in the target document in turn, and perform the following text storage steps on each section of the text to be processed: Detecting the text type of the currently read text to be processed, the text type including: title or text; When it is detected that the current text to be processed is a title, obtaining the title level corresponding to the title; Detecting whether there is data in the text information of the last saved data item in the storage medium; the data item indicates the correspondence between the title information and the text information, the title information in each of the data items is a title saved in sequence according to the title level, and the text information in each of the data items is the text corresponding to the title with the lowest level in the corresponding title information; If there is data in the text information, the data item is stored in a resource pool, the text information of the data item and data less than or equal to the title level in the title information are cleared, the title is written into a corresponding position in the title information, and a detection step is performed; If there is no data in the text information, writing the title into the corresponding position in the title information and executing the detection step; When it is detected that the current text to be processed is a main text, the current text to be processed is stored at the position of the main text information in the data item, and a detection step is performed; The detection step comprises: Detect whether the target document has been read. If not, re-execute the above text storage step for the next section of text to be processed. If it has been read, store the data item in the resource pool.

[0007] In a second aspect, the present application provides a text segmentation device for a RAG system, the text segmentation device for a RAG system comprising: A text receiving module, used for receiving a target document; The storage module is used to read each section of the to-be-processed text in the target document in sequence, and perform the following text storage steps on each section of the to-be-processed text: Detecting the text type of the currently read text to be processed, the text type including: title or text; When it is detected that the current text to be processed is a title, obtaining the title level corresponding to the title; Detecting whether there is data in the text information of the last saved data item in the storage medium; the data item indicates the correspondence between the title information and the text information, the title information in each of the data items is a title saved in sequence according to the title level, and the text information in each of the data items is the text corresponding to the title with the lowest level in the corresponding title information; If there is data in the text information, the data item is stored in a resource pool, the text information of the data item and data less than or equal to the title level in the title information are cleared, the title is written into a corresponding position in the title information, and a detection step is performed; If there is no data in the text information, writing the title into the corresponding position in the title information and executing the detection step; When it is detected that the current text to be processed is a main text, the current text to be processed is stored at the position of the main text information in the data item, and a detection step is performed; The detection step comprises: Detect whether the target document has been read. If not, re-execute the above text storage step for the next section of text to be processed. If it has been read, store the data item in the resource pool.

[0008] In a third aspect, the present application provides a text query method for a RAG system, the method applying a resource pool constructed by the text segmentation method for the RAG system as described in the first aspect, the method comprising: Receive inquiries; Acquire corresponding data items from the resource pool according to the query question; Combining the text information in the data item according to the title information and the text information in the data item to obtain an answer corresponding to the query question; Output the answer.

[0009] In a fourth aspect, the present application provides a text query device of a RAG system, wherein the text query device of the RAG system comprises: A question receiving module, used for receiving query questions; A data acquisition module, used to acquire corresponding data items from the resource pool according to the query question; An answer acquisition module, used to combine the text information in the data item according to the title information and the text information in the data item to obtain an answer corresponding to the query question; The answer output module is used to output the answer.

[0010] According to the specific embodiments provided in this application, this application discloses the following technical effects: The present application provides a text segmentation and query method and device for a RAG system, the method comprising: receiving a target document; reading each section of the to-be-processed text in the target document in sequence, and performing the following text storage steps for each section of the to-be-processed text: detecting the text type of the current to-be-processed text read, the text type including: title or text; when detecting that the current to-be-processed text is a title, obtaining the title level corresponding to the title; detecting whether the text information in the data item last saved in the storage medium has data; the data item indicates the correspondence between the title information and the text information, the title information in each data item is the title saved in sequence according to the title level, and the text information in each data item is the corresponding title information intermediate level The text corresponding to the lowest title; if there is data in the text information, the data item is stored in the resource pool, the text information of the data item and the data less than or equal to the title level in the title information are cleared, the title is written into the corresponding position in the title information, and the detection step is performed; if there is no data in the text information, the title is written into the corresponding position in the title information, and the detection step is performed; when it is detected that the current text to be processed is the main text, the current text to be processed is stored in the position of the main text information in the data item, and the detection step is performed; the detection step includes: detecting whether the target document has been read, if not, re-execute the above text storage step for the next paragraph of the text to be processed, if it has been read, store the data item in the resource pool. In the present disclosure, the text is split according to the title granularity, the data between the two titles represents a paragraph, and the paragraph title and the superior title are also added to increase the retrievable information of the paragraph, and retain the relevant information to the greatest extent, so that the relevant text can be found more accurately during the retrieval, avoiding the loss of the text, and improving the completeness and accuracy of the answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 The following is a flow chart of a text segmentation method for a RAG system according to an exemplary embodiment. Figure 1 ; Figure 2 is a schematic diagram of a data structure of a title cache according to an exemplary embodiment; Figure 3 The following is a flow chart of a text segmentation method for a RAG system according to an exemplary embodiment. Figure 2 ; Figure 4is a functional module diagram of a text segmentation device for a RAG system according to an exemplary embodiment; Figure 5 is a flowchart of a text query method of a RAG system according to an exemplary embodiment; Figure 6 is a functional module diagram of a text query device of a RAG system according to an exemplary embodiment; Figure 7 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0015] In recent years, breakthroughs in the field of deep learning have made it possible to build large-scale pre-trained language models, such as BERT and GPT. These models can capture rich language structure and semantic information by performing self-supervised learning on a large amount of unlabeled text. However, despite the powerful generation capabilities of these models, they still have some limitations, such as: Knowledge limitations: The knowledge of pre-trained models is usually limited to the time range of their training data and cannot be updated with new knowledge in real time.

[0016] Factual bias: Due to bias or deficiencies in the training data, the model may generate inaccurate or false factual information.

[0017] Fabrication Bias: The model tends to generate fabricated content, especially in the absence of specific context.

[0018] In order to overcome the above limitations, researchers began to explore how to enable language models to use external knowledge bases to enhance their generation capabilities. This requires the model to be able to retrieve the most relevant content from massive information and incorporate it into the generated text. This demand has led to the development of RAG technology.

[0019] Currently, the RAG system uses the following text segmentation methods to generate vector databases: The first method: directly use the five text segmentation methods that come with Langchain. The general method is as follows: 1) Convert the document into text through different types of file processing programs such as Word reader, Orc reader, Markdown reader, etc. 2) Input the text into the text segmentation tool; 3) The text segmentation tool divides the text into text blocks and stores them into the vector database.

[0020] However, this segmentation method has the following disadvantages: 1) Improper segmentation: If fixed-length segmentation is used, two closely related texts will be split into two paragraphs. It is possible that one paragraph cannot be retrieved during the query, resulting in problems with the answer.

[0021] 2) Slow efficiency: If the model is used for segmentation, it will seriously affect the efficiency of data writing and query.

[0022] 3) Segmentation accuracy: Models are usually trained in English. The semantics and grammar of Chinese are different from those of English, which ultimately leads to inaccurate segmented data.

[0023] The second method is to use punctuation marks and the features of the upper and lower texts for segmentation. The usual method is as follows: 1. Obtain the first text T1 of the paragraph to be segmented, and clean the first text T1 of the paragraph to be segmented to obtain the second text T2 of the paragraph to be segmented; 2. According to the punctuation marks of the second text T2 of the paragraph to be segmented, the second text T2 of the paragraph to be segmented is segmented to generate a sentence list; the sentence list includes: a plurality of sentences formed by the segmentation of the second text T2 of the paragraph to be segmented; 3. Traverse each sentence in the sentence list, process each sentence in the sentence list, and construct a paragraph list; the paragraph list includes: a plurality of new paragraphs obtained after processing each sentence in the sentence list; 4. Traverse the paragraph list, clean each new paragraph in the paragraph list, and generate a paragraph pool.

[0024] However, this technical solution has the following disadvantages: 1) Documents use irregular punctuation: When a document starts with a bullet point, and the previous paragraph may be separated by a period, the segmentation may be unreasonable, that is, a paragraph with the same meaning may be split into two, causing problems when searching; 2) Semantic segmentation problem: Sentences are segmented only by punctuation marks, so all relevant data cannot be retrieved during query. For example, the following text: A Hospital's regulations on employee working hours Article 1: These Regulations are formulated in accordance with the relevant provisions of Law B in order to reasonably arrange the working hours and rest time of employees, safeguard the rest rights of employees, mobilize the enthusiasm of employees and promote the development of the cause.

[0025] Article 2: These regulations apply to employees of state organs, social groups, enterprises, institutions and other organizations within the territory of Country C.

[0026] Article 3: Employees work 8 hours a day and 40 hours a week.

[0027] According to the second method, the first, second and third items will each generate a paragraph. When searching, if the user enters the question "What are the working hours regulations", the first and third items can be searched, but the second item cannot be searched. This is because the second item has little relevance to the query question.

[0028] In order to solve the above technical problems, the present disclosure proposes a text segmentation and query method and device for a RAG system.

[0029] Figure 1 is a flowchart of a text segmentation method for a RAG system according to an exemplary embodiment. Figure 1 As shown, the method includes the following steps S101-S108: In step S101, a target document is received.

[0030] Each section of the text to be processed in the target document is read in sequence, and the following text storage steps S102-S107 are performed on each section of the text to be processed: In step S102, the text type of the currently read text to be processed is detected, and the text type includes: title or text.

[0031] After receiving the target document, the target document may be parsed by a corresponding document parser to obtain the format of the target document.

[0032] Document formats may include: word processing document format, spreadsheet document format, presentation document format, and portable document format.

[0033] Among them, word processing document formats may include: DOC / DOCX, TXT, and RTF (Rich Text Format); Spreadsheet document formats can include: XLS / XLSX and CSV (Comma-Separated Values); Presentation document formats can include: PPT / PPTX and ODP (Open Document Presentation); Portable document formats may include: PDF (Portable Document Format).

[0034] After receiving the target document and identifying the document format of the target document, each section of the text to be processed in the target document is read in turn. After reading the current text to be processed, the text type of the current text to be processed needs to be identified first, that is, it needs to be identified whether the current text to be processed is a title or a body text. There will be different processing solutions based on the title and the body text.

[0035] In step S103, when it is detected that the current text to be processed is a title, the title level corresponding to the title is obtained.

[0036] In a document, the heading level is mainly used to organize the hierarchical structure of the content, so that readers can quickly understand the general content and key points of the document. Different document formats and editing software have different ways of setting the heading level, but the basic principles are similar.

[0037] First-level headings (usually correspond to the highest level of the document outline).

[0038] Style features: The font is usually large and bold, and it stands out on the page. It is often a direct reflection of the document theme and is used to summarize the core content of the entire document. For example, in an academic paper, the first-level title may be "On the Application of Artificial Intelligence in the Medical Field", which clarifies the scope of the paper.

[0039] Layout location: Usually located at the beginning of the document or at the start of a new chapter. There may be a large gap between it and the main text to enhance visual distinction. In some document formats (such as markdown), the first-level title is usually marked with the "#" symbol, such as "#Application of artificial intelligence in the medical field".

[0040] Function: Helps readers quickly grasp the key points when browsing the document and determine whether the document is relevant to their needs. At the same time, it is also an important basis for building a document directory. Many word processing software can automatically generate a document directory based on the first-level title.

[0041] Secondary heading.

[0042] Style features: The font size is between the first-level title and the main text, and may also be bold. It is a subdivision of the first-level title content, used to explain the main sub-content under the topic. Taking the academic paper just now as an example, the second-level title may be "Application of Artificial Intelligence in Disease Diagnosis", which is an important branch under the big topic of "Application of Artificial Intelligence in the Medical Field".

[0043] Layout position: After the first-level title, the content of the first-level title is usually followed, and there is a certain distance between the first-level title and the text to reflect the hierarchy. In markdown, "##" is often used to mark the second-level title, such as "## Application of artificial intelligence in disease diagnosis".

[0044] Function: Further refine the document content so that readers can understand the document structure more clearly. For documents with long length and complex content, the secondary heading can guide readers to quickly locate the sub-content section of interest.

[0045] Level 3 headings and below.

[0046] Style features: The font size and style will decrease in accordance with the overall style and level of the document. Some software may keep the font bold, but the degree will gradually decrease. These titles are a more detailed division of the content of the upper-level title. For example, under the second-level title "Application of artificial intelligence in disease diagnosis", the third-level title can be "Disease imaging diagnosis based on deep learning", which more specifically describes an application method in the sub-field of disease diagnosis.

[0047] Layout position: nested in the content of the upper-level title, and maintain appropriate spacing with the upper-level title and the text to which it belongs. In markdown, the third-level title is marked with "###", such as "###Disease imaging diagnosis based on deep learning".

[0048] Function: With the continuous subdivision of heading levels, complex document content can be broken down into small, easy-to-understand parts, making it easier for authors to write and readers to read. For documents with rich content such as technical documents and long reports, lower-level headings help build a detailed knowledge system and make the document content more organized.

[0049] Here are some ways to identify heading levels in a document: 1. Identify through format features.

[0050] 1. Font and font size.

[0051] Generally speaking, the higher the title level, the larger the font size. The first-level title usually uses the largest font size. For example, in many documents, the first-level title may be in size 2 or size 2 font, while the second-level title may be in size 3 font, and the third-level title is in size 4 font, etc. The title font is often more eye-catching, and may be bold or black. For example, in a company report, the first-level title "Annual Work Summary" is bold and has a larger font size, while the second-level title "Sales Performance Analysis" is slightly smaller and also bold. By comparing the font size and bold features, the title level can be distinguished.

[0052] 2. Color and style.

[0053] Some documents use colors to distinguish the heading levels. For example, the first-level heading may be a dark color such as black, and the second-level heading may be a lighter color such as gray. At the same time, the headings may also have some special styles, such as adding underlines, shadows, or using different fonts (such as Songti for the first-level heading and Kaiti for the second-level heading) to distinguish them. However, this situation is relatively rare because too many color and style changes may make the document look messy.

[0054] 3. Paragraph format.

[0055] The paragraph spacing between the title and the text can be used as a basis for identification. The first-level title usually has a larger spacing from the text, followed by the second-level title, and the third-level title has an even smaller spacing. In addition, the paragraph indents of the title itself may also be different. For example, the first-level title may be centered, the second-level title may be left-aligned and have a certain indentation, and the third-level title may have more indentation. Taking an academic paper as an example, the first-level title "Introduction" is centered on the page, with more blank lines between it and the text below; the second-level title "Research Background" is left-aligned and has a slightly smaller spacing from the text below; the third-level title "Current Status of Domestic and Foreign Research" is indented and has a smaller spacing from the text.

[0056] 2. Identify through document structure and content logic.

[0057] 1. Content hierarchy relationship.

[0058] Observing the scope of content covered by the title can help identify the title level. The first-level title is usually a high-level summary of the entire document topic and has the broadest scope. The second-level title is a breakdown of the topic covered by the first-level title, with a narrower scope. For example, in a product manual, the first-level title is "Product Overview", and the second-level titles "Product Features Introduction" and "Product Technical Parameters" are specific content parts broken down from the large topic of product overview. The title level can be determined by the inclusion relationship of this content scope.

[0059] 2. Numbering system.

[0060] Some documents use numbers to indicate the level of headings. For example, a numbering system such as "1., 1.1, 1.1.1" is used. "1." usually represents a first-level heading, "1.1" represents a second-level heading, and "1.1.1" represents a third-level heading. This numbering method can intuitively reflect the hierarchical relationship between headings. For example, in a project plan, "1. Project goals" is a first-level heading, "1.1 Short-term goals" is a second-level heading, and "1.1.1 First-stage goals" is a third-level heading.

[0061] 3. Grammatical and vocabulary characteristics.

[0062] First-level titles are often more abstract and general, and the wording is general. Second-level titles are relatively specific, and third-level titles are more detailed. For example, the first-level title is "The Impact of Education Reform", the second-level title is "The Impact on Teaching Methods", and the third-level title is "The Specific Application of New Teaching Methods in the Classroom". The wording of the title shows the level change from broad to specific, which helps to identify the title level.

[0063] 3. Use software functions to identify.

[0064] 1. Outline view (in word processing software).

[0065] Many word processing software have an outline view function. In outline view, the software will automatically display the headings in a hierarchical manner according to the heading levels of the document. The first-level heading will be at the highest level, the second-level heading will be at the next level, and so on. Users can clearly see the hierarchy of headings through outline view, and can easily adjust the heading levels.

[0066] 2. Style function (in word processing software).

[0067] The style function in the software can help identify the heading level. When writing a document, the author may use the software's preset styles or custom styles to set the heading. By looking at the style name (such as "Title 1", "Title 2", etc.) or style attributes (such as font, font size, etc.), you can determine the heading level. For example, in word processing software, the "Title 1" style is usually used for the first-level heading, and the "Title 2" style is used for the second-level heading. By looking at which style is applied to the heading, you can accurately identify the heading level.

[0068] The present disclosure may identify the title level through any of the above methods.

[0069] In step S104, it is detected whether there is data in the text information of the data item last saved in the storage medium; the data item indicates the correspondence between the title information and the text information, the title information in each data item is the title saved in sequence according to the title level, and the text information in each data item is the text corresponding to the lowest level title in the corresponding title information.

[0070] The format of the data items will be pre-set in the storage medium, as shown in Table 1:

[0071] In Table 1, 4 levels of titles are preset, that is, when the recognized title level is level 1, it is filled in the position corresponding to the level 1 title, and when the recognized title level is level 2, it is filled in the position corresponding to the level 2 title.

[0072] In step S105, if there is data in the text information, the data item is stored in the resource pool, the text information of the data item and the data less than or equal to the title level in the title information are cleared, the title is written to the corresponding position in the title information, and the detection step of step S108 is executed, that is, whether the target document has been read. If not, the above text storage step is re-executed for the next section of text to be processed. If it has been read, the data item is stored in the resource pool.

[0073] After identifying the level of the title, it is also necessary to detect whether there is data in the text information of the previous data item in the storage medium. If there is data in the text information of the last saved data item, it indicates that the last identification was the text corresponding to the previous title. Moreover, since the identification this time is the title, it indicates that the text corresponding to the previous title has been identified and needs to be stored in the resource pool for subsequent retrieval. After saving the last data item to the resource pool, clear the text information of the data item and the data in the title information that is less than or equal to the title level, and write the title to the corresponding position in the title information.

[0074] Take the following target document as an example: 1.AAA.

[0075] 1.1BB.

[0076] Bbb.

[0077] 1.2CC.

[0078] Ccc.

[0079] Among them, 1.AAA is a level 1 title, 1.1BB is a level 2 title, Bbb is the text corresponding to level 2 title BB, 1.2CC is also a level 2 title, Ccc is the text corresponding to level 2 title CC, 1.AAA is the first paragraph of text to be processed, 1.1BB is the second paragraph of text to be processed, Bbb is the third paragraph of text to be processed, 1.2CC is the fourth paragraph of text to be processed, and Ccc is the fifth paragraph of text to be processed.

[0080] The data items in the storage medium are shown in Table 2:

[0081] Assume that the fourth paragraph of text to be processed is currently recognized: 1.2CC.

[0082] Identify 1.2CC as a title, and the title level is 2. Detect that the text information in the last saved data item (Table 2) in the storage medium has data, then save Table 2 to the resource pool, clear the text information of the data item and the data less than or equal to the title level in the title information, and get Table 3:

[0083] Write CC into the position corresponding to the level 2 title to obtain Table 4:

[0084] In one embodiment, storing the data item in the resource pool includes the following sub-steps A1-A5: A1. Check whether the number of characters in the text information in the data item is greater than a preset number of characters.

[0085] A2. If the number of characters is greater than a preset number, the characters in the text message are divided into a plurality of text blocks according to the preset number.

[0086] A3. Divide the data item into a corresponding number of sub-data items according to the number of text blocks, each sub-data item includes: title information and current text block information, and the current text block information includes: current text block, total number of text blocks and order of current text block.

[0087] A4. Store the sub-data items in the resource pool.

[0088] A5. If it is not greater than the preset number of characters, the data item is stored in the resource pool.

[0089] The sub-data items are shown in Table 5:

[0090] Take Table 6 as an example for explanation:

[0091] When it is necessary to save Table 6 to the resource pool, the main text 111222333444555 can be divided into multiple text blocks. Assuming the preset number is 3, the main text 111222333444555 is divided into 5 text blocks in order, namely: 111, 222, 333, 444 and 555, among which 111 is ranked 1, 222 is ranked 2, 333 is ranked 3, 444 is ranked 4, and 555 is ranked 5, and the 5 sub-data items shown in Table 7 are obtained, and then these 5 sub-data items are stored in the resource pool respectively.

[0092]

[0093] In one embodiment, the current text block information may further include: the last preset number of characters of the previous text block.

[0094] The sub-data items may also be as shown in Table 8, in which the last two characters of the previous text block are added:

[0095] Continuing to use Table 6 as an example, when Table 6 needs to be saved to the resource pool, the sub-data items obtained at this time are shown in Table 9:

[0096] Furthermore, the data items may be vectorized and then stored in the resource pool, and only the title information and the current text block may be vectorized.

[0097] In order to ensure that the vector can express semantics as much as possible, the vectorized text needs to be kept within a certain range. Therefore, if the paragraph is too large, it needs to be split, and then the order of the text blocks and the title information need to be retained after the split.

[0098] In step S106, if there is no data in the text information, the title is written to the corresponding position in the title information, and the detection step of step SS108 is executed, that is, whether the target document has been read. If not, the above text storage step is re-executed for the next segment of text to be processed. If it has been read, the data item is stored in the resource pool.

[0099] Specifically, writing the title into the corresponding position in the title information includes the following sub-steps B1-B3: B1. Check whether there is data in the title information; B2. If there is no data in the title information, write the title into the corresponding position in the title information; B3. If there is data in the title information, clear the data in the title information of the data item that is less than or equal to the title level, and write the title into the corresponding position in the title information.

[0100] If there is no data in the text information, it means that the last recognition was a title. Then the title is written to the corresponding position in the title information. If data has been written to the position with the same title level as the title in the data item, it is also necessary to delete the position with the same title level as the title and the title at a lower level, and write the title to the corresponding position in the title information.

[0101] Take the following target document as an example: 1.AAA.

[0102] 2.DDD.

[0103] Among them, 1.AAA is a level 1 title with no text, and 2.DDD is a level 1 title.

[0104] The data items are shown in Table 10:

[0105] Assume that the second paragraph of text to be processed is currently recognized: 2.DDD.

[0106] 2. DDD is identified as a title, and the title level is level 1. It is detected that the text information in the last saved data item (Table 5) in the storage medium has no data, so the data AAA corresponding to the level 1 title is cleared, and DDD is filled into the position corresponding to the level 1 title, and Table 11 is obtained:

[0107] In step S107, when it is detected that the current text to be processed is the main text, the current text to be processed is stored in the location of the main text information in the data item, and the detection step of step S108 is executed, that is, it is detected whether the target document has been read. If it has not been read, the above text storage step is re-executed for the next section of text to be processed. If it has been read, the data item is stored in the resource pool.

[0108] If it is detected that the current text to be processed is the main text, it means that this is the content after the last recognized content. If the last recognized content was the title, it is the main text corresponding to the title. If the last recognized content was the main text, this time it is the next paragraph of the main text. The current text to be processed is stored in the position of the main text information in the data item.

[0109] Take the following target document as an example: 1.AAA.

[0110] Aaaa.

[0111] Ddd.

[0112] Among them, 1.AAA is the level 1 title, Aaa is the first paragraph of text corresponding to the level 1 title, and Ddd is the second paragraph of text corresponding to the level 1 title.

[0113] The data items are shown in Table 12:

[0114] Assume that the third paragraph of text to be processed is currently recognized: Ddd.

[0115] Identify Ddd as text, detect that the text information in the last saved data item (Table 2) in the storage medium has data, then write Ddd to the position corresponding to the text, and get Table 13:

[0116] In step S108, it is detected whether the target document has been read. If not, the above text storage step is re-executed for the next section of text to be processed. If it has been read, the data item is stored in the resource pool.

[0117] In the present disclosure, text is split according to the title granularity, and the data between two titles represents a paragraph. At the same time, the paragraph title and the parent title are added to increase the searchable information of the paragraph and retain relevant information to the greatest extent, so that relevant text can be found more accurately during retrieval and the loss of text can be avoided.

[0118] Since the text is split based on the title as the granularity, the semantic loss between texts caused by fixed length is avoided, and the present disclosure can accurately recognize Chinese, thereby avoiding the disadvantage that some models have low support for Chinese.

[0119] In one achievable manner, the storage medium in the present disclosure is a cache.

[0120] Cache is the ability of a computer to temporarily store frequently accessed data or calculation results to reduce latency and improve performance.

[0121] The cache of header information is as follows Figure 2 The data structure shown in the figure is implemented by using an array in a computer data structure. Figure 2 The level 1 title is in the first position of the array, the level 2 title is in the second position of the array, and so on. When clearing titles after the nth level, add 1 to the index and set the current value to empty; when querying titles of a specified level, you can directly get the corresponding title according to the level.

[0122] The cache of the text information can also add corresponding end characters at the end of the text when it is actually used.

[0123] When the document is first recognized, the cache of the title information and the cache of the body information may also be initialized.

[0124] Figure 3 is a flowchart of a text segmentation method for a RAG system according to an exemplary embodiment. Figure 3 As shown, the basic steps of the entire technical solution for text segmentation are as follows: 1> Document parsing: Parse the document through the corresponding text parser.

[0125] 2> Initialize the title cache and initialize the body cache.

[0126] The title information cache mainly stores the actual title content corresponding to the first-level title, second-level title, etc. When the title changes, each level of title will change dynamically. The text cache saves the entire paragraph with the same semantics. The format of the title cache is <title level: title content>.

[0127] The title cache here represents the title information in the above embodiment, and the text cache here represents the text information in the above embodiment.

[0128] 3> Read a piece of text: determine whether the text is the main text or the title.

[0129] 3.1> If it is a title.

[0130] 3.1.1> Check whether there is content in the text cache.

[0131] If the body cache has content, execute 3.1.1.1; if the body cache has no content, execute 3.1.2.

[0132] 3.1.1.1> Split the text in the body cache into text blocks according to the specified length, [the last 20 characters of the previous text block: the current text block: the total number of text blocks: the order of the current text block].

[0133] The text is divided into text blocks of specified size, and each block is defined according to the following rules: First-level title -> Second-level title -> Third-level title -> … -> N-level title: The last 20 characters of the previous text block: The current text block: The total number of text blocks: The sorting of the current text block.

[0134] 3.1.1.2> Save the title information and text block together and store them in the vector database in the form of [title information: text information].

[0135] The title information here is: first-level title -> second-level title -> third-level title ->… -> n-level title; the text information here is: the last 20 characters of the previous text block: the current text block: the total number of text blocks: the sorting of the current text block.

[0136] The part to be vectorized is the first-level title -> second-level title -> third-level title -> … -> n-level title: the last 20 characters of the previous text block: the current text block.

[0137] The total number of text blocks and the order of the current text block are not vectorized.

[0138] 3.1.2> If it is detected that there is no content in the text cache, then the text cache will be directly cleared, the titles that are less than or equal to the level will be cleared, and the corresponding title information will be updated.

[0139] 3.2>If it is text: add the current text to the text cache.

[0140] 4> Determine whether there is any text.

[0141] 4.1> If there is still text, jump back to 3.

[0142] 4.2>If there is no text, jump to 5.

[0143] 5> Determine whether there is content in the text cache.

[0144] 5.1>If there is content.

[0145] 5.1.1> Divide the text in the body cache into text blocks according to the specified length, [the last 20 characters of the previous text block: the current text block: the total number of text blocks: the order of the current text block].

[0146] 5.1.2>After the definition is completed, the title information and the text block are saved together and stored in the vector database in the form of [title information: text information].

[0147] The title information here is: first-level title -> second-level title -> third-level title ->… -> n-level title; the text information here is: the last 20 characters of the previous text block: the current text block: the total number of text blocks: the sorting of the current text block.

[0148] The part to be vectorized is the first-level title -> second-level title -> third-level title -> … -> n-level title: the last 20 characters of the previous text block: the current text block.

[0149] The total number of text blocks and the order of the current text block are not vectorized.

[0150] 5.1.3> Jump to 6.

[0151] 5.2>If there is no content, jump to 6.

[0152] 6>End.

[0153] By adopting the method disclosed in the present invention, a complete and accurate knowledge base can be established, providing a good foundation for future enterprise-level knowledge retrieval.

[0154] Based on the same inventive concept, the embodiment of the present application also provides a text segmentation device for a RAG system for implementing the above-mentioned text segmentation method for a RAG system. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more embodiments of the text segmentation device for a RAG system provided below can refer to the limitations of the text segmentation method for a RAG system above, and will not be repeated here.

[0155] In an exemplary embodiment, Figure 4 As shown, a text segmentation device for a RAG system is provided, comprising: A text receiving module 11, used for receiving a target document; The storage module 12 is used to read each section of the text to be processed in the target document in sequence, and perform the following text storage steps on each section of the text to be processed: Detect the text type of the currently read text to be processed, the text type includes: title or text; When it is detected that the current text to be processed is a title, the title level corresponding to the title is obtained; Detect whether there is data in the text information of the last saved data item in the storage medium; the data item indicates the corresponding relationship between the title information and the text information, the title information in each data item is the title saved in sequence according to the title level, and the text information in each data item is the text corresponding to the title with the lowest level in the corresponding title information; If there is data in the text information, the data item is stored in the resource pool, the text information of the data item and the data less than or equal to the title level in the title information are cleared, and the title is written into the corresponding position in the title information; If there is no data in the text information, writing the title into the corresponding position in the title information; When it is detected that the current text to be processed is a main text, the current text to be processed is stored at the position of the main text information in the data item; Detect whether the target document has been read. If not, re-execute the above text storage step for the next section of text to be processed. If it has been read, store the data item in the resource pool.

[0156] In one embodiment, in terms of writing the title into a corresponding position in the title information, the storage module 12 is specifically configured to: Detecting whether there is data in the title information; If there is no data in the title information, writing the title into a corresponding position in the title information; If there is data in the title information, clear the data in the title information of the data item that is less than or equal to the title level, and write the title into a corresponding position in the title information.

[0157] In one embodiment, in the aspect of storing the data item in the resource pool, the storage module 12 is specifically used for: Detecting whether the number of characters in the text information in the data item is greater than a preset number of characters; If the number of characters is greater than the preset number, the characters in the text message are divided into a plurality of text blocks according to the preset number; Dividing the data item into a corresponding number of sub-data items according to the number of the text blocks, each of the sub-data items includes: the title information and current text block information, the current text block information includes: the current text block, the total number of the text blocks and the order of the current text block; storing the sub-data item in the resource pool; If it is not greater than the preset number of characters, the data item is stored in the resource pool.

[0158] In one embodiment, the current text block information further includes: a preset number of characters at the end of the previous text block.

[0159] In one embodiment, in the aspect of storing the data item in the resource pool, the storage module 12 is specifically used for: The data items are vectorized and stored in the resource pool.

[0160] In one embodiment, the storage medium is a cache.

[0161] Figure 5 is a flowchart of a text query method of a RAG system according to an exemplary embodiment. Figure 5 As shown, the method applies a resource pool constructed by a text segmentation method for a RAG system in any of the above embodiments, and the method includes the following sub-steps S201-S204: S201, receiving a query question.

[0162] S202: Obtain corresponding data items from a resource pool according to the query question.

[0163] After receiving the user's query question, based on the question, the corresponding data item is obtained from the resource pool. After obtaining the data item.

[0164] S203: Combine the text information in the data item according to the title information and text information in the data item to obtain an answer corresponding to the query question.

[0165] After the corresponding data items are found in the resource pool, the data items are concatenated to obtain the answer corresponding to the query question.

[0166] S204. Output the answer.

[0167] In the present disclosure, the RAG system includes a RAG module and an LLM module. The LLM module receives a query question and sends the question to the RAG module. The RAG module obtains the corresponding data item from the resource pool according to the query question, obtains the answer corresponding to the query question according to the data item, and then sends the answer to the LLM module. The LLM module outputs the answer.

[0168] In one embodiment, combining the text information in the data item according to the title information and the text information in the data item to obtain the answer corresponding to the query question includes the following sub-steps C1-C5: C1. Grouping the acquired data items according to title information and text information.

[0169] C2. Detecting whether the text information is missing according to the total number of text blocks in the text information and the order of the current text block.

[0170] C3. If yes, search the resource pool again for the target data item corresponding to the missing information according to the title information, the total number of the text blocks and the order of the current text block.

[0171] The queried data items are grouped according to the title information and text information, and a judgment is made based on the total number of text blocks of each text and the order of each text block to find out whether there are any missing text blocks. If there are missing text blocks, a search is performed again based on the last preset number of characters of the previous text block, the title information, and the order of the text blocks to find the data items corresponding to the missing information.

[0172] C4. Concatenate the information in the data item and the target data item to obtain an answer corresponding to the query question.

[0173] C5. If not, combining the text information in the data item according to the title information and text information in the data item to obtain an answer corresponding to the query question.

[0174] In one embodiment, obtaining corresponding data items from the resource pool according to the query question includes the following sub-steps D1-D2: D1. Vectorize the query question to obtain a vectorized question.

[0175] D2. Obtain corresponding data items from the resource pool according to the vectorized problem.

[0176] After receiving the query question, the query question may be vectorized, and a query may be performed in the resource pool according to the vectorized vector of the question.

[0177] When merging, you need to combine according to the information retained when splitting to ensure the order of the paragraphs, so as to prevent LLM from giving them to you in a disordered state after checking them out, which may cause LLM to misunderstand and give wrong steps. At the same time, when combining, you also need to complete the missing parts to ensure that no part of the information is lost when LLM answers.

[0178] Based on the same inventive concept, the embodiment of the present application also provides a text query device for a RAG system for implementing the text query method of the RAG system involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more text query device embodiments of the RAG system provided below can refer to the limitations of the text query method of the RAG system above, and will not be repeated here.

[0179] In an exemplary embodiment, Figure 6 As shown, a text query device of a RAG system is provided, comprising: The question receiving module 21 is used to receive query questions.

[0180] The data acquisition module 22 is used to acquire corresponding data items from the resource pool according to the query question.

[0181] The answer acquisition module 23 is used to combine the text information in the data item according to the title information and the text information in the data item to obtain the answer corresponding to the query question.

[0182] The answer output module 24 is used to output the answer.

[0183] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a text query method of a RAG system is implemented.

[0184] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0185] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0186] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0187] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0188] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0189] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0190] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0191] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0192] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A text segmentation method for a RAG system, characterized in that: The text segmentation method for the RAG system comprises: receiving the target document; Read each section of the text to be processed in the target document in sequence, and perform the following text storage steps on each section of the text to be processed: Detecting the text type of the currently read text to be processed, the text type including: title or text; When it is detected that the current text to be processed is a title, obtaining the title level corresponding to the title; Detecting whether there is data in the text information of the last saved data item in the storage medium; the data item indicates the correspondence between the title information and the text information, the title information in each of the data items is a title saved in sequence according to the title level, and the text information in each of the data items is the text corresponding to the title with the lowest level in the corresponding title information; If there is data in the text information, the data item is stored in a resource pool, the text information of the data item and data less than or equal to the title level in the title information are cleared, the title is written into a corresponding position in the title information, and a detection step is performed; If there is no data in the text information, writing the title into the corresponding position in the title information and executing the detection step; When it is detected that the current text to be processed is a main text, the current text to be processed is stored at the position of the main text information in the data item, and the detection step is performed; The detection step comprises: Detect whether the target document has been read. If not, re-execute the above text storage step for the next section of text to be processed. If it has been read, store the data item in the resource pool.

2. The text segmentation method for RAG system according to claim 1, characterized in that: The step of writing the title into a corresponding position in the title information includes: Detecting whether there is data in the title information; If there is no data in the title information, writing the title into a corresponding position in the title information; If there is data in the title information, clear the data in the title information that is less than or equal to the title level, and write the title into the corresponding position in the title information.

3. The text segmentation method for RAG system according to claim 1, characterized in that: The storing the data item in a resource pool includes: Detecting whether the number of characters in the text information in the data item is greater than a preset number of characters; If the number of characters is greater than the preset number, the characters in the text message are divided into a plurality of text blocks according to the preset number; Dividing the data item into a corresponding number of sub-data items according to the number of the text blocks, each of the sub-data items includes: the title information and current text block information, the current text block information includes: the current text block, the total number of the text blocks and the order of the current text block; storing the sub-data item in the resource pool; If it is not greater than the preset number of characters, the data item is stored in the resource pool.

4. The text segmentation method for RAG system according to claim 3, characterized in that: The current text block information also includes: the last preset number of characters of the previous text block.

5. The text segmentation method for RAG system according to claim 3, characterized in that: The storing the data item in a resource pool comprises: The data items are vectorized and stored in the resource pool.

6. The text segmentation method for RAG system according to claim 1, characterized in that: The storage medium is a cache.

7. A text segmentation device for a RAG system, characterized in that: The text segmentation device for the RAG system comprises: A text receiving module, used for receiving a target document; The storage module is used to read each section of the to-be-processed text in the target document in sequence, and perform the following text storage steps on each section of the to-be-processed text: Detecting the text type of the currently read text to be processed, the text type including: title or text; When it is detected that the current text to be processed is a title, obtaining the title level corresponding to the title; Detecting whether there is data in the text information of the last saved data item in the storage medium; the data item indicates the correspondence between the title information and the text information, the title information in each of the data items is a title saved in sequence according to the title level, and the text information in each of the data items is the text corresponding to the title with the lowest level in the corresponding title information; If there is data in the text information, the data item is stored in a resource pool, the text information of the data item and data less than or equal to the title level in the title information are cleared, the title is written into a corresponding position in the title information, and a detection step is performed; If there is no data in the text information, writing the title into the corresponding position in the title information and executing the detection step; When it is detected that the current text to be processed is a main text, the current text to be processed is stored at the position of the main text information in the data item, and the detection step is performed; The detection step comprises: Detect whether the target document has been read. If not, re-execute the above text storage step for the next section of text to be processed. If it has been read, store the data item in the resource pool.

8. A text query method for a RAG system, characterized in that: The method applies a resource pool constructed by the text segmentation method for a RAG system according to any one of claims 1 to 6, and the method comprises: Receive inquiries; Acquire corresponding data items from the resource pool according to the query question; Combining the text information in the data item according to the title information and the text information in the data item to obtain an answer corresponding to the query question; Output the answer.

9. The text query method of the RAG system according to claim 8, characterized in that: The combining the text information in the data item according to the title information and the text information in the data item to obtain the answer corresponding to the query question includes: Grouping the acquired data items according to title information and text information; Detecting whether the text information is missing according to the total number of text blocks in the text information and the order of the current text block; If yes, searching the resource pool for the target data item corresponding to the missing information again according to the title information, the total number of the text blocks and the order of the current text block; Joining the information in the data item and the target data item to obtain an answer corresponding to the query question; If not, the text information in the data item is combined according to the title information and the text information in the data item to obtain the answer corresponding to the query question.

10. A text query device for a RAG system, characterized in that: The text query device of the RAG system comprises: A question receiving module, used for receiving query questions; A data acquisition module, used to acquire corresponding data items from a resource pool according to the query question; An answer acquisition module, used to combine the text information in the data item according to the title information and the text information in the data item to obtain an answer corresponding to the query question; The answer output module is used to output the answer.

Images