Knowledge base construction method of RAG system, electronic equipment and storage medium

Abstract

The invention provides a knowledge base construction method of an RAG system, electronic equipment and a storage medium, and relates to the technical field of big data. The method comprises the following steps: analyzing an original knowledge document according to the format of the original knowledge document of the urban rail transit industry to generate a lightweight markup language text; the lightweight markup language text is split according to the multi-level chapter titles and the text content, a knowledge tree is generated, and each leaf node of the knowledge tree is a segment of text content; the knowledge base of the urban rail transit industry comprises a plurality of segments of text content under each level of title, a plurality of segments of text content under each level of title are combined and segmented according to a preset slice threshold value and a preset parent text block threshold value, a plurality of blocks are generated, an urban rail professional dictionary is adopted, entity recognition and relation extraction are conducted on the blocks, and a knowledge graph of an original knowledge document is constructed. And the knowledge base of the urban rail transit industry is used for enabling the RAG system to perform knowledge retrieval. The professional knowledge base can be automatically constructed, and the manual maintenance cost is reduced.

Description

Technical Field

[0001] This application relates to the field of big data technology, and more specifically, to a method for constructing a knowledge base for a RAG system, an electronic device, and a storage medium. Background Technology

[0002] Intelligent question answering technology based on Retrieval-Augmented Generation (RAG) technology relies heavily on the quality of data parsing. With the rapid development of the urban rail transit industry, urban rail engineering professionals have accumulated a massive amount of technical documents, standards, maintenance manuals, and other professional knowledge materials.

[0003] Currently, the industry mainly manages professional knowledge materials through traditional document management systems, keyword-based retrieval systems, and early knowledge graph systems. Traditional document management systems only provide simple document storage and retrieval functions, resulting in low retrieval efficiency. Keyword-based retrieval systems can only perform simple text matching and cannot handle synonymous expressions and semantic relationships of professional terms. Early knowledge graph systems require a large amount of manual annotation and relationship building, resulting in high maintenance costs and difficulty in adapting to the rapid updates of urban rail transit engineering professional knowledge. Summary of the Invention

[0004] The purpose of this application is to address the shortcomings of the prior art by providing a knowledge base construction method, electronic device, and storage medium for a RAG system, so as to automate the construction of a professional knowledge base and reduce manual maintenance costs.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a method for constructing a knowledge base for a RAG system, the method comprising: Based on the format of the original knowledge document in the urban rail transit industry, the original knowledge document is parsed to generate lightweight markup language text; The lightweight markup language text is split according to the multi-level chapter titles and text content to generate a knowledge tree, where each leaf node of the knowledge tree is a piece of text content. Based on the preset slicing threshold and the preset parent text block threshold, multiple text contents under each level of heading are combined and segmented to generate multiple blocks. The multiple blocks include: a first number of text blocks and a second number of parent text blocks. Each parent text block is a combination of at least two text blocks. Each block has the chapter titles of each level to which the target segment text content belongs. Using a pre-set urban rail transit professional dictionary, entity recognition and relationship extraction are performed on the multiple blocks to construct a knowledge graph of the original knowledge document. The knowledge base of the urban rail transit industry includes the knowledge graph, and the knowledge base of the urban rail transit industry is used to enable the RAG system to perform knowledge retrieval.

[0006] Optionally, the step of combining and segmenting multiple text segments under each heading level according to a preset slicing threshold and a preset parent text block threshold to generate multiple blocks includes: Based on the length of the multiple text segments and the preset slicing threshold, the multiple text segments are combined and segmented to generate the first number of text blocks; Generate a parent text block based on a segment of text content that is divided into at least two text blocks; The parent text block and the other paragraph text content under each level heading are combined sequentially. The parent text block is updated according to the combined text length and the preset parent text block threshold to obtain the second number of parent text blocks. Based on the target segment text content corresponding to each text block or each parent text block, determine the chapter titles of each level to which the target segment text content belongs from the knowledge tree; Add corresponding chapter titles at each level to each text block and each parent text block to generate each sub-block.

[0007] Optionally, the step of combining and segmenting the multiple text segments according to their lengths and the preset slicing threshold to generate the first number of text blocks includes: If the length of the first text segment is less than the preset slicing threshold, and the length of the first text segment combined with the second text segment is also less than the preset slicing threshold, continue to combine with other text segments until the length of the last combination is greater than or equal to the preset slicing threshold. A text block is generated based on the text content of the previous combination of the last one; If the length of the first text segment is less than the preset slicing threshold, and the length of the combination of the first text segment and the second text segment is greater than or equal to the preset slicing threshold, it is determined that the first text segment generates a text block, and the second text segment and the third text segment are combined for judgment. If the length of the first text segment is greater than or equal to the preset slicing threshold, the first text segment is segmented to generate at least two text blocks.

[0008] Optionally, the step of sequentially combining the parent text block and the other paragraph text content under each level heading, and updating the parent text block according to the combined text length and the preset parent text block threshold, includes: The parent text block and other paragraphs of text content under each level heading are combined sequentially. For each combination of text content, it is determined whether the length of the combined text is greater than or equal to the preset parent text block threshold. If the length of the text after each combination is less than the preset parent text block threshold, the parent text block is updated to the length of the text after each combination. If the length of the text after each combination is greater than or equal to the preset slice threshold, then the parent text block remains unchanged.

[0009] Optionally, the step of using a preset urban rail transit professional dictionary to perform entity recognition and relation extraction on the multiple blocks to construct a knowledge graph of the original knowledge document includes: Dependency analysis is performed on the statements in each block to determine the syntactic relations of the statements in each block; Based on the entity information in the urban rail transit professional dictionary, entity extraction is performed on each block to determine multiple entities in each block; Based on the relational information in the urban rail transit professional dictionary, the syntactic relations of the statements in each block, and the multiple entities in each block, relational extraction is performed on each block to determine the entity relations of the multiple entities in each block. Based on the attribute information in the urban rail transit professional dictionary and the multiple entities in each block, determine the attribute information of the multiple entities in each block; The knowledge graph is generated based on the entity relationships and attribute information of multiple entities in each block.

[0010] Optionally, the step of parsing the original knowledge document according to the format of the original knowledge document in the urban rail transit industry to generate lightweight markup language text includes: If the original knowledge document is in PDF format, perform layout detection on the original knowledge document to determine the content of the first table, the content of the first text block, and / or the content of the first image in the original knowledge document; The content of the first table is recognized and converted into a Hypertext Markup Language format to generate a first JSON table; The text and formulas in the first text block are identified respectively, and the first text content and the formulas in the preset layout are generated. The content of the first image is stored, and a first index identifier for the storage file of the image content is generated; The lightweight markup language text is generated based on the first JSON table, the first text content, the preset formatting formula, and / or the first index identifier.

[0011] Optionally, the step of parsing the original knowledge document according to the format of the original knowledge document in the urban rail transit industry to generate lightweight markup language text includes: If the original knowledge document is in docx format, the original knowledge document is parsed to determine the content of the second table, the content of the second text block, and / or the content of the second image in the original knowledge document; The content of the second table is split using preset rules to generate a second JSON table; Retrieve the second text content from the second text block; The content of the second image is stored, and a second index identifier for the storage file of the second image content is generated; The lightweight markup language text is generated based on the second JSON table, the second text content, and / or the second index identifier.

[0012] Optionally, the method further includes: Generate a text summary based on the text content under the preset hierarchical chapter headings; and / or, Generate table slice content based on the table summary, table title, and chapter title; and / or, The multiple blocks are processed into text vectors to generate multiple text vectors; the knowledge base of the urban rail transit industry also includes: the text summary, the table slice content and / or the multiple text vectors.

[0013] Secondly, embodiments of this application provide a knowledge base construction apparatus for a RAG system, the apparatus comprising: The document parsing module is used to parse the original knowledge document according to the format of the original knowledge document in the urban rail transit industry and generate lightweight markup language text. The knowledge tree construction module is used to split the lightweight markup language text according to the multi-level chapter titles and text content to generate a knowledge tree, wherein each leaf node of the knowledge tree is a piece of text content; The block generation module is used to combine and segment multiple text contents under each level of heading according to a preset slicing threshold and a preset parent text block threshold to generate multiple blocks. The multiple blocks include: a first number of text blocks and a second number of parent text blocks. Each parent text block is a combination of at least two text blocks. Each block has the chapter titles of each level to which the target segment text content belongs. The knowledge graph construction module is used to construct the knowledge graph of the original knowledge document by using a preset urban rail transit professional dictionary to perform entity recognition and relation extraction on the multiple blocks. The knowledge base of the urban rail transit industry includes the knowledge graph, and the knowledge base of the urban rail transit industry is used to enable the RAG system to perform knowledge retrieval.

[0014] Optionally, the block generation module is specifically used to combine and segment the multiple text segments according to their lengths and the preset slicing threshold to generate the first number of text blocks; generate a parent text block based on a segment of text content that has been segmented into at least two text blocks; sequentially combine the parent text block with other segments of text content under each level of heading, and update the parent text block according to the combined text length and the preset parent text block threshold to obtain the second number of parent text blocks; determine the level of chapter headings to which the target segment text content belongs from the knowledge tree based on the target segment text content corresponding to each text block or each parent text block; add corresponding level of chapter headings to each text block and each parent text block to generate each block.

[0015] Optionally, the block generation module is further configured to: if the length of the first text segment is less than the preset slicing threshold, and the length of the combination of the first and second text segments is also less than the preset slicing threshold, continue to combine with other text segments until the length of the last combination is greater than or equal to the preset slicing threshold; generate a text block based on the text content of the previous combination; if the length of the first text segment is less than the preset slicing threshold, and the length of the combination of the first and second text segments is greater than or equal to the preset slicing threshold, determine that the first text segment generates a text block, and perform a combination judgment on the second and third text segments; if the length of the first text segment is greater than or equal to the preset slicing threshold, segment the first text segment to generate at least two text blocks.

[0016] Optionally, the block generation module is further configured to sequentially combine the parent text block and other paragraph text content under each level heading, and after each combination of text content, determine whether the length of the text after each combination is greater than or equal to the preset parent text block threshold; if the length of the text after each combination is less than the preset parent text block threshold, update the parent text block to the length of the text after each combination; if the length of the text after each combination is greater than or equal to the preset slicing threshold, determine that the parent text block remains unchanged.

[0017] Optionally, the knowledge graph construction module is specifically used to perform dependency analysis on the statements in each block to determine the syntactic relations of the statements in each block; extract entities from each block based on the entity information in the urban rail transit professional dictionary to determine multiple entities in each block; extract relations from each block based on the relation information in the urban rail transit professional dictionary, the syntactic relations of the statements in each block, and the multiple entities in each block to determine the entity relations of the multiple entities in each block; determine the attribute information of the multiple entities in each block based on the attribute information in the urban rail transit professional dictionary and the multiple entities in each block; and generate the knowledge graph based on the entity relations of the multiple entities in each block and the attribute information of the multiple entities in each block.

[0018] Optionally, the document parsing module is specifically configured to: if the original knowledge document is in PDF format, perform layout detection on the original knowledge document to determine the first table content, the first text block content, and / or the first image content in the original knowledge document; recognize the first table content and perform Hypertext Markup Language format conversion to generate a first JSON table; recognize the text and formulas in the first text block content to generate first text content and a preset-formatted formula; store the first image content to generate a first index identifier for the storage file of the image content; and generate the lightweight markup language text based on the first JSON table, the first text content, the preset-formatted formula, and / or the first index identifier.

[0019] Optionally, the document parsing module is specifically used to parse the original knowledge document if the original knowledge document is in docx format, determine the second table content, the second text block content, and / or the second image content in the original knowledge document; split the second table content according to preset rules to generate a second JSON table; obtain the second text content in the second text block content; store the second image content and generate a second index identifier for the storage file of the second image content; and generate the lightweight markup language text according to the second JSON table, the second text content, and / or the second index identifier.

[0020] Optionally, the device further includes: The information generation module is used to generate text summaries based on the text content under the preset hierarchical chapter titles; and / or to generate table slice content based on the table summary, table title, and the chapter title to which the table belongs; and / or to perform text vectorization processing on the multiple slices to generate multiple text vectors; the knowledge base of the urban rail transit industry also includes: the text summaries, the table slice content, and / or the multiple text vectors.

[0021] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the steps of the knowledge base construction method of the RAG system as described in any of the first aspects.

[0022] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the knowledge base construction method of the RAG system as described in any of the first aspects.

[0023] The beneficial effects of this application are: The knowledge base construction method, electronic device, and storage medium of the RAG system provided in this application effectively preserve the hierarchical structure and semantic information of the original knowledge documents by converting them into lightweight markup language text, thereby improving the data quality in subsequent processing stages. By combining preset slicing thresholds and parent text block thresholds, the text content under each level of heading is intelligently combined and segmented to generate ordinary text blocks and parent text blocks. The parent text blocks, as the upper-level summary content of the ordinary text blocks, can significantly improve the understanding of complex questions and the logical coherence of answer generation during the RAG retrieval stage. By generating knowledge graphs through entity recognition and relation extraction, the professional knowledge base can be automatically constructed, reducing manual maintenance costs and greatly enhancing the intelligent question-answering capabilities of the RAG system. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating the knowledge base construction method provided in this application embodiment. Figure 1 ; Figure 2 A flowchart illustrating the knowledge base construction method provided in this application embodiment. Figure 2 ; Figure 3 A flowchart illustrating the knowledge base construction method provided in this application embodiment. Figure 3 ; Figure 4 A flowchart illustrating the knowledge base construction method provided in this application embodiment. Figure 4 ; Figure 5 A document format conversion flowchart provided for embodiments of this application; Figure 6 A flowchart illustrating the knowledge base construction method provided in this application embodiment. Figure 5 ; Figure 7 This is a schematic diagram of the knowledge graph construction apparatus provided in the embodiments of this application; Figure 8 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0027] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0028] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0030] Figure 1 A flowchart illustrating the knowledge base construction method provided in this application embodiment. Figure 1 ,like Figure 1 As shown, the method may include: S101. Based on the format of the original knowledge document in the urban rail transit industry, parse the original knowledge document and generate lightweight markup language text.

[0031] In this embodiment, the urban rail transit industry's urban rail engineering professionals have accumulated a massive amount of professional knowledge data. The original knowledge documents corresponding to this professional knowledge data are stored in the urban rail professional database and file server. The original knowledge documents cover the design schemes, maintenance manuals, technical specifications, etc. of the urban rail engineering profession, and the document formats of different original knowledge documents may be different.

[0032] To better recognize the content of original knowledge documents in different document formats, it is necessary to first parse the content of these documents and generate lightweight markup language text. Different parsing methods are required for different document formats. Specifically, the format of the original knowledge document can be identified based on its file extension; for example, a document with the extension ".pdf" is recognized as a PDF document, and a document with the extension ".docx" is recognized as a Word document.

[0033] In some embodiments, the original knowledge document is converted into a plain text document, such as a TXT text document, according to the original knowledge document's format. A preset regular expression library defines the conversion methods for different levels of chapter titles and text structures. The preset regular expression library is used to convert the TXT text document to obtain a lightweight markup language document in Markdown format.

[0034] S102. Based on the multi-level chapter titles and text content, the lightweight markup language text is split to generate a knowledge tree, where each leaf node of the knowledge tree is a piece of text content.

[0035] In this embodiment, a knowledge tree is generated based on the relationship between multi-level chapter titles and text content in the Markdown format text. The root node of the knowledge tree can be the title or identifier of the original knowledge document. The child nodes of the knowledge tree are chapter titles. The parent node of each child node is the parent chapter title of the corresponding chapter title of each child node. The next-level child node of each child node is the next-level chapter title of the corresponding chapter title of each child node. Each leaf node of the knowledge tree is the text content corresponding to each paragraph. If a first-level chapter title contains multiple paragraphs, then the child nodes of that chapter title include multiple leaf nodes.

[0036] In some embodiments, if the original knowledge document includes list items, the child nodes of the knowledge tree may also include list items. The leaf nodes under the child nodes of the list items are the contents of the list items. The list items may be ordered lists or unordered lists. An ordered list consists of multiple contents that begin with a number, while an unordered list consists of multiple contents that begin with a non-numeric symbol such as "·".

[0037] It should be noted that if there is at least one piece of text under a first-level chapter title, and there is a next-level chapter title after the first-level chapter title, then the first-level chapter title is a child node, the at least one piece of text is at least one leaf node under the child node, the next-level chapter title is the next-level child node under the child node, and the leaf nodes of the next-level child node are the pieces of text under the next-level chapter title.

[0038] S103. Based on the preset slicing threshold and the preset parent text block threshold, combine and segment the multiple text contents under each level of heading to generate multiple blocks. The multiple blocks include: a first number of text blocks and a second number of parent text blocks. Each parent text block is a combination of at least two text blocks. Each block has the chapter titles of each level to which the target text content belongs.

[0039] In this embodiment, the knowledge tree is traversed starting from the root node. Multiple text segments belonging to the same child node are identified within leaf nodes. These segments are considered to be under the same level of chapter heading. Following the order of the text segments, and based on the token count of each segment, the segments are sequentially combined until the total token count of the combined text exceeds a preset slicing threshold. This ensures that the total token count of the combined text does not exceed the preset slicing threshold. The combined text is treated as a single text block. The token count represents the number of basic units within each text segment, which can be a single character or a word. The token count can be either the number of characters or words in each text segment.

[0040] For text content whose token count is greater than the preset slicing threshold even without being combined, the text content can be divided into at least two text blocks so that the token count of each text block is less than the preset slicing threshold. During the process of slicing the text content, the text content can be split according to the period in the text content to ensure the semantic integrity of the split text blocks.

[0041] Among them, the text blocks corresponding to text content that cannot be combined with other text content, the text blocks corresponding to combined text content, and the text blocks split from text content together constitute the first number of text blocks.

[0042] Furthermore, to ensure semantic integrity, for a piece of text content that is divided into at least two text blocks, a parent text block can be generated based on the text content of that piece of text content or the combination of the text content of that piece of text content with other text content. Whether the text content of that piece of text content is combined with other text content is determined by whether the number of tokens after the text content of that piece of text content is combined with other text content is less than or equal to a preset threshold for the parent text block.

[0043] Specifically, if the number of tokens after combining the text content with other text content is less than or equal to the preset parent text block threshold, then the text content is combined with other text content into a parent text block. If the number of tokens after combining the text content with other text content is greater than the preset parent text block threshold, then the text content is treated as a separate parent text block. For each text content segmented into at least two text blocks, the above method is used to generate parent text blocks to obtain the second number of parent text blocks.

[0044] In some embodiments, the preset slice threshold and the preset parent text block threshold can be set according to the structural information of the original knowledge document, such as the number of chapters and the complexity of the content. This embodiment does not impose any restrictions on this.

[0045] S104. Using a pre-set urban rail transit professional dictionary, entity recognition and relation extraction are performed on multiple blocks to construct a knowledge graph of the original knowledge documents. The knowledge base of the urban rail transit industry includes: knowledge graph. The knowledge base of the urban rail transit industry is used to enable the RAG system to perform knowledge retrieval.

[0046] In this embodiment, the urban rail transit professional dictionary pre-summarizes multiple entities and relationships involved in the industrial business of the urban rail transit industry. For the multiple blocks generated in S103 above, the multiple entities and relationships in the urban rail transit professional dictionary are used to perform entity recognition and relationship extraction on the content of each block, determine the entities contained in each block and the relationships between the entities, and construct a knowledge graph containing entity triples based on the entities and the relationships between the entities. The entity triples include a head entity, a relationship, and a tail entity.

[0047] For example, entities can include: equipment and facilities, components and materials, defects and malfunctions, operations and processes, personnel and organizations, standards and specifications, and location information. Relationships can include: ontology hierarchy relationships, such as "...is a...", "...is a part of...", attribute relationships, such as "...has...", "...is... located in...", and event relationships, such as "...acts on...", "...causes...", etc.

[0048] In some embodiments, the urban rail transit professional dictionary may also include entity attributes. The entity attributes in the urban rail transit professional dictionary are used to identify the entity attributes of the content of each block, and to determine the entity attribute pair of each entity. The entity attribute pair contains the entity, the attribute, and the attribute value, and to construct a knowledge graph containing entity triples, entity attribute pairs of the head entity, and entity attribute pairs of the tail entity.

[0049] If the entity attribute information of an entity is not described in a block, the constructed knowledge graph may not contain entity attribute pairs.

[0050] For example, for the block content "Milling operation J20231027 processed rail-G107", through entity recognition and relation extraction, the entity triple (milling operation J20231027, operation, rail-G107) is determined. The entity attribute pairs include: (milling operation J20231027, operation time, 2023-10-27), (milling operation J20231027, operating equipment, milling vehicle), (rail-G107, number, G107), (rail-G107, length, 500 meters).

[0051] For example, the specific application of knowledge graphs in RAG can be as follows: When a user asks a question, the system first performs semantic parsing to identify the entities and relationships in the question. Based on the entities and relationships in the question, it retrieves entity triples containing the entity and relationship, as well as entity attribute pairs related to the entity triples, from the knowledge graph. Based on the entity triples and entity attribute pairs, it determines the answer to the question.

[0052] The knowledge base construction method of the RAG system provided in the above embodiments can effectively preserve the hierarchical structure and semantic information of the original knowledge documents by converting them into lightweight markup language text, thereby improving the data quality of subsequent processing stages. By combining preset slicing thresholds and parent text block thresholds, the text content under each level of heading is intelligently combined and segmented to generate ordinary text blocks and parent text blocks. The parent text block, as the upper-level summary content of the ordinary text block, can significantly improve the understanding of complex questions and the logical coherence of answer generation during the RAG retrieval stage. By generating a knowledge graph through entity recognition and relation extraction, the professional knowledge base can be automatically constructed, reducing manual maintenance costs and greatly enhancing the intelligent question-answering capability of the RAG system.

[0053] In one possible implementation, Figure 2 A flowchart illustrating the knowledge base construction method provided in this application embodiment. Figure 2 ,like Figure 2 As shown, the process of combining and segmenting multiple text segments under each heading level to generate multiple blocks based on preset slicing thresholds and preset parent text block thresholds in S103 may include: S201. Based on the length of multiple text segments and the preset slicing threshold, combine and segment the multiple text segments to generate a first number of text blocks.

[0054] In this embodiment, for multiple text segments under each chapter title, the segments are combined based on their length (e.g., the number of tokens). Combining stops when the combined text length exceeds a preset slicing threshold, ensuring the combined text length is less than or equal to the threshold. If the combined length of two text segments exceeds the threshold, they are not combined. During the combination process, multiple text segments must be combined sequentially; combinations across paragraphs are not allowed. For example, if the combined length of the first and second text segments exceeds the threshold, they are not combined; the first segment remains a separate text block, while the second and third segments are combined.

[0055] If the length of a text segment already exceeds a preset slicing threshold, the text segment needs to be split into at least two text blocks, each with a length less than or equal to the preset slicing threshold. The text segment can be split based on the period (or period) in the text segment.

[0056] S202. Generate a parent text block based on a piece of text content that has been divided into at least two text blocks.

[0057] In this embodiment, for a piece of text content whose length is greater than a preset slicing threshold, after being divided into at least two text blocks, the piece of text content is used as a parent text block.

[0058] S203. Combine a parent text block with the text content of other paragraphs under each level heading in sequence. Update a parent text block according to the length of the combined text and the preset threshold of the parent text block to obtain the second number of parent text blocks.

[0059] In this embodiment, the multiple text segments under each heading level include: a text segment corresponding to each parent text block and other text segments excluding that text segment. If there are multiple parent text blocks in the multiple text segments under each heading level, for each parent text block, all text segments except the text segment corresponding to that parent text block are other text segments corresponding to that parent text block. For example, if each heading level contains text content 1-text content 5, where text content 1 and text content 3 are parent text block 1 and parent text block 2 respectively, then the other paragraphs corresponding to parent text block 1 are text content 2-text content 5, and the other paragraphs corresponding to parent text block 2 are text content 1, 2, 4, and 5.

[0060] Determine if the length of the parent text block is greater than or equal to a preset parent text block threshold. If the length of the parent text block is greater than or equal to the preset parent text block threshold, then do not update the parent text block.

[0061] If the length of the parent text block is less than the preset parent text block threshold, the parent text block and an adjacent text segment are combined. Based on the length of the combined text and the preset parent text block threshold, it is determined whether to update the parent text block. If the parent text block is updated, it is updated to be a combination of the original parent text block and an adjacent text segment.

[0062] S204. Based on the target segment text content of each text block or each parent text block, determine the chapter titles of each level to which the target segment text content belongs from the knowledge tree.

[0063] In this embodiment, in order to better determine the scope of knowledge in RAG knowledge Q&A, chapter titles can be added to each text block or each parent text block.

[0064] Specifically, for each text block or the target segment text content corresponding to each parent text block, the lowest-level chapter title to which the target segment text content belongs is determined from the knowledge tree, and the chapter titles at each level are determined by indexing upwards from the lowest-level chapter title.

[0065] S205. Add corresponding chapter titles at each level to each text block and each parent text block to generate each sub-block.

[0066] In this embodiment, for example, the name of the original knowledge document is A, the first-level headings are A1, A2, ..., the second-level headings are A11, A12, ..., A21, A22, ..., the third-level headings are A111, A112, ..., A121, A122, ..., A211, A212, ..., A221, A222, ..., if the lowest-level chapter heading of a text block or its parent text block is A111, then the chapter headings at each level are A-A1-A11-A111. Corresponding chapter headings at each level are added to each text block and each parent text block to generate each sub-block.

[0067] After generating blocks by adding corresponding chapter titles at all levels to each text block and each parent text block, when performing entity recognition and relation extraction on the blocks to generate a knowledge graph, the knowledge graph retains the corresponding chapter titles at all levels for each entity triple and entity attribute pair.

[0068] The knowledge base construction method provided in the above embodiments combines and segments multiple text content according to a preset slicing threshold, ensuring that the length of each text block is appropriate, that is, not too long leading to information redundancy, nor too short leading to information incompleteness. By generating a parent text block from the text content that has been segmented into at least two text blocks, and combining the parent text block with other text content to update the parent text block, not only is the semantic coherence of the original text preserved, but the integrity and logic of the parent text block in terms of content are also ensured. This helps to improve the accuracy and contextual understanding ability of the RAG system when answering complex questions. Adding corresponding chapter titles at all levels to each text block and parent text block clarifies the position of the text block and parent text block in their respective documents, providing rich contextual information, enabling the RAG system to better understand and reference relevant background knowledge when retrieving and answering questions.

[0069] In one possible implementation, the process of combining and segmenting multiple text segments based on their lengths and a preset slicing threshold to generate a first number of text blocks may include: If the length of the first text segment is less than the preset slicing threshold, and the length of the first text segment combined with the second text segment is also less than the preset slicing threshold, continue to combine with other text segments until the length of the last combination is greater than or equal to the preset slicing threshold; generate a text block based on the text content of the previous combination after the last combination; if the length of the first text segment is less than the preset slicing threshold, and the length of the first text segment combined with the second text segment is greater than or equal to the preset slicing threshold, determine that the first text segment generates a text block, and perform a combination judgment on the second and third text segments; if the length of the first text segment is greater than or equal to the preset slicing threshold, segment the fifth text segment to generate at least two text blocks.

[0070] It should be noted that the first, second, and third paragraphs of text mentioned above are only for distinguishing texts of different lengths, and do not mean that there are three paragraphs of text under a first-level chapter title.

[0071] In this embodiment, for multiple text segments under a first-level chapter title, the process starts from the first text segment and traverses the text segment to determine whether the length of the first text segment is less than a preset slicing threshold. If the length of the first text segment is less than the preset slicing threshold, the first text segment and the adjacent second text segment are combined, and it is determined whether the length of the combined text segment is also less than the preset slicing threshold.

[0072] If the combined length of the first and adjacent second text segments is less than a preset slicing threshold, the first and second text segments are combined. The combined text segments are then combined with the third text segment, and a judgment is made. If the combined length with the third text segment is greater than or equal to the preset slicing threshold, the text segments are not combined with the third text segment, and a text block is generated from the combined text segments of the first and second text segments. If the combined length with the third text segment is less than the preset slicing threshold, the text segments are then combined with the fourth text segment, and a judgment is made.

[0073] If the combined length of the first and adjacent second text segments is greater than or equal to the preset slicing threshold, then the first and second text segments are not combined. Instead, a text block is generated based on the first text segment, and the second and third text segments are combined and judged.

[0074] If the content of the first text segment is greater than or equal to the preset slicing threshold, then the first text segment is sliced ​​to generate at least two text blocks.

[0075] The knowledge base construction method provided in the above embodiments combines and segments multiple text contents according to a preset slicing threshold to ensure that the length of each text block is appropriate, that is, it is not too long to cause information redundancy, nor too short to cause information incompleteness.

[0076] In one possible implementation, the process of sequentially combining a parent text block with the text content of other paragraphs under each heading in S203, and updating a parent text block based on the length of the combined text and a preset threshold for the parent text block, may include: The text content of a parent text block and other paragraphs under each level heading are combined sequentially. After each combination, it is determined whether the length of the combined text is greater than or equal to the preset parent text block threshold. If the length of the combined text is less than the preset parent text block threshold, the parent text block is updated to the length of the combined text. If the length of the combined text is greater than or equal to the preset slicing threshold, the parent text block is determined to remain unchanged.

[0077] In this embodiment, it is determined whether the length of the parent text block is greater than or equal to a preset parent text block threshold. If the length of the parent text block is greater than or equal to the preset parent text block threshold, the parent text block is not updated. If the length of the parent text block is less than the preset parent text block threshold, the parent text block and adjacent segment text content are combined. If the length of the combined text is less than the preset parent text block threshold, the parent text block is updated to the combined text content, and the process of combining with other segment text content continues and is evaluated. If the length of the combined text is greater than or equal to the preset parent text block threshold, the parent text block remains unchanged.

[0078] For example, under chapter title A111 there are three text segments with word counts of 350, 100, and 550 respectively. If the preset slicing threshold is 512 and the parent text block threshold is 1024, then when combining and segmenting the text content, the combined word count of text content 1 and text content 2 is 450, and the combined word count of text content 1, text content 2, and text content 3 is 1000. Compared with the preset slicing threshold, the first two segments combined are less than 512, and the three segments combined are greater than 512. Therefore, text content 1 and text content 2 are combined into one text block. If the number of characters in text content 3 (550) is greater than the preset slicing threshold (512), then text content 3 is divided into two text blocks, and text content 3 is used as a parent text block. The number of characters in this parent text block (550) is less than the parent text block threshold (1024). The number of characters in the combination of this parent text block and text content 2 is 900, which is less than the parent text block threshold (1024). It is then combined with text content 1, and the number of characters in the combination is 1000. Then, the parent text block is updated to be a combination of text content 1, text content 2, and text content 3.

[0079] The knowledge base construction method provided in the above embodiments generates a parent text block by dividing the text content into at least two text blocks, and updates the parent text block by combining the parent text block with other segments of text content. This not only preserves the semantic coherence of the original text, but also ensures the integrity and logic of the parent text block in terms of content, which helps to improve the accuracy and contextual understanding ability of the RAG system when solving complex problems.

[0080] In one possible implementation, Figure 3 A flowchart illustrating the knowledge base construction method provided in this application embodiment. Figure 3 ,like Figure 3 As shown, the process of S104 above, which uses a pre-set urban rail transit professional dictionary to perform entity recognition and relation extraction on multiple blocks to construct a knowledge graph of the original knowledge document, may include: S301. Perform dependency analysis on the statements in each block to determine the syntactic relations of the statements in each block.

[0081] S302. Based on the entity information in the urban rail transit professional dictionary, extract entities from each block and determine multiple entities for each block.

[0082] S303. Based on the relational information in the urban rail transit professional dictionary, the syntactic relations of the statements in each block, and the multiple entities in each block, perform relation extraction on each block to determine the entity relations of the multiple entities in each block.

[0083] S304. Based on the attribute information in the urban rail transit professional dictionary and the multiple entities in each block, determine the attribute information of the multiple entities in each block.

[0084] S305. Generate a knowledge graph based on the entity relationships and attribute information of multiple entities in each block.

[0085] In this embodiment, Natural Language Processing (NLP) tools are used to analyze each sentence in each block sequentially to determine the grammar of each sentence. The grammar of each sentence includes the part of speech of each word in the sentence, such as subject, predicate, object, attributive, adverbial, etc. Multiple entities from the urban rail transit professional dictionary are used to perform entity recognition on the content of each block to determine the multiple entities contained in each block. Multiple relations and grammars from the urban rail transit professional dictionary are used to extract relations from the multiple entities. The relationships between entities are determined based on the part of speech of each word. Based on the entities and the relationships between entities, entity triples are constructed.

[0086] The urban rail transit professional dictionary can also include entity attributes. By using the entity attributes in the urban rail transit professional dictionary, entity attribute identification is performed on the content of each block to determine the entity attribute pair for each entity. The entity attribute pair contains entity, attribute, and attribute value, and a knowledge graph containing entity triples, entity attribute pairs of head entities, and entity attribute pairs of tail entities is constructed.

[0087] In this process, dependency analysis is equivalent to syntactic analysis, and relation extraction is equivalent to semantic analysis. For example, for the sentence "A high-efficiency milling machine can completely repair severe rail corrugation," dependency analysis can determine that "milling machine" and "repair" are subject-verb grammar, "repair" and "corrugation" are verb-object grammar, and "high-efficiency" modifies "milling machine," "severe," and "rail" modifies "corrugation," forming a noun-head relation. Through entity recognition, "milling machine" and "rail" are identified as equipment, "corrugation" as a defect, and "repair" as a process. Relation extraction is performed on the sentence, and the subject-verb and verb-object relations are determined through "repair," forming the triple (milling machine, can repair, rail corrugation). Similarly, the triple (corrugation, related component, rail) can also be determined. Through attribute analysis, the noun-head relation of each entity is searched to determine the attribute pairs of each entity, namely (milling machine, attribute, high-efficiency) and (corrugation, attribute, severe).

[0088] The knowledge base construction method of the RAG system provided in the above embodiments generates a knowledge graph through dependency analysis, entity recognition, and relation extraction, thereby realizing the automated construction of a professional knowledge base, reducing manual maintenance costs, and greatly enhancing the intelligent question answering capability of the RAG system.

[0089] In one possible implementation, Figure 4 A flowchart illustrating the knowledge base construction method provided in this application embodiment. Figure 4 ,like Figure 4As shown, the process of S101 above, which parses the original knowledge document according to the format of the original knowledge document in the urban rail transit industry and generates lightweight markup language text, may include: S401. If the original knowledge document is in PDF format, perform layout detection on the original knowledge document to determine the content of the first table, the content of the first text block, and / or the content of the first image in the original knowledge document.

[0090] S402. Recognize the content of the first table and convert it to Hypertext Markup Language format to generate the first JSON table.

[0091] S403. Recognize the text and formulas in the first text block and generate the first text content and the preset formatted formulas.

[0092] S404. Store the content of the first image and generate the first index identifier of the storage file of the image content.

[0093] S405. Generate lightweight markup language text based on the first JSON table, the first text content, the preset formatted formulas, and / or the first index identifier.

[0094] In this embodiment, for example, Figure 5 The document format conversion flowchart provided for the embodiments of this application is as follows: Figure 5 As shown, the document formats are divided into PDF, doc, and docx Word files. For the original knowledge document in PDF format, the layout analysis tool is first launched to identify the type of each part of the original knowledge document in PDF format, and to identify the content of the first table, the content of the first text block, and / or the content of the first image.

[0095] For the identified first table content, such as urban rail equipment parameter tables or maintenance cycle tables, the table's row and column structure and content information are first extracted using a table structure recognition tool. Then, a table structure detection and recognition model is used to segment the content based on its logical relationships and data volume. For example, when the first table content contains multiple different types of equipment parameters, the table structure detection and recognition model will split it into multiple independent smaller tables to avoid reduced efficiency and difficulty in information extraction during subsequent searches due to excessively long tables.

[0096] For multiple independent small tables, a first JSON table in JSON format is generated through HyperText Markup Language (HTML) format conversion.

[0097] For the first identified text content, a formula detection model is used to check if it contains formulas. If formulas are found, a formula recognition model is used to convert them into LaTeX format for storage. This format conversion is beneficial for subsequent large models to accurately understand and process the formula content. For example, in the analysis of urban rail transit track mechanics calculation formulas, LaTeX format can clearly present various parameters and calculation relationships. For the text portion of the first text content other than formulas, optical character recognition (OCR) is used to identify the first text content.

[0098] For the first image content identified, such as urban rail line layout diagrams and equipment structure diagrams, it is uploaded to the object storage system, and a unique first index identifier, i.e., image ID, is generated at the corresponding text location in the document. This ID is associated with the image's storage path in the object storage so that the image can be quickly located and retrieved in subsequent searches.

[0099] The first JSON table, the first text content, the pre-formatted formulas, and / or the first index identifier are combined according to their positions in the original knowledge document to generate a TXT file. A regular expression parsing library is then used to convert the TXT file into a Markdown file.

[0100] In one possible implementation, Figure 6 A flowchart illustrating the knowledge base construction method provided in this application embodiment. Figure 5 ,like Figure 6 As shown, the process of S101 above, which parses the original knowledge document according to the format of the original knowledge document in the urban rail transit industry and generates lightweight markup language text, may include: S501. If the original knowledge document is in docx format, parse the original knowledge document to determine the content of the second table, the content of the second text block, and / or the content of the second image in the original knowledge document.

[0101] S502. Split the content of the second table using preset rules to generate a second JSON table.

[0102] S503. Obtain the second text content from the second text block.

[0103] S504. Store the content of the second image and generate a second index identifier for the storage file of the second image content.

[0104] S505. Generate lightweight markup language text based on the second JSON table, the second text content, and / or the second index identifier.

[0105] In this embodiment, for the original knowledge document in docx format, the python-docx parsing library of Python is used to parse the original knowledge document and extract the content of the second table, the content of the second text block and / or the content of the second image.

[0106] The extracted content of the second table is processed using a DataFrame format. The table is then split according to a predefined splitting logic, such as by table theme or data category. For example, a table containing operational data for various urban rail lines can be split into multiple sub-tables based on line names.

[0107] The methods for generating the second JSON table and the second index identifier are the same as those in S402 and S404 above. Since the original knowledge document is itself a document format, the second text content can be directly extracted. The process of generating lightweight markup language text in S605 above is the same as the process of generating lightweight markup language text in S405 above, and will not be described in detail here.

[0108] Furthermore, such as Figure 5 As shown, for original knowledge documents in doc format, one method can be used to convert them to docx format, and then convert them to lightweight markup language text using the methods described in S601-S605 above. Alternatively, one method can be used to directly use the tika parsing library to obtain the text content and generate a TXT document. This embodiment does not impose any restrictions on this method.

[0109] The knowledge base construction method provided in the above embodiments can accurately identify and structure text, formulas, tables, and images in original knowledge documents of different formats, facilitating subsequent identification of their content.

[0110] In one possible implementation, the method may further include: Based on the text content under the preset hierarchical chapter titles, generate text summaries; and / or, based on the table summary, table title, and the chapter title to which the table belongs, generate table slice content; and / or, perform text vectorization processing on multiple blocks to generate multiple text vectors; the knowledge base of the urban rail transit industry also includes: text summaries, table slice content, and / or multiple text vectors.

[0111] In this embodiment, based on the title hierarchy of the lightweight markup language text, a large model is called to generate text summaries for the text content corresponding to chapter titles below a preset level. The length of the text summaries is strictly controlled within the range of a preset slicing threshold.

[0112] For table content in lightweight markup language text, the large model is called to comprehensively describe the table content, generate a table summary, and combine the table summary with the table title and related chapter titles to form table slice content.

[0113] Multiple text blocks are input into an embedding model optimized for the urban rail transit industry for vectorization, resulting in multiple text vectors for each block. These text vectors are then stored in an Elasticsearch vector database. Elasticsearch's high-performance retrieval and powerful distributed processing capabilities meet the storage and rapid retrieval needs of large amounts of vector data in urban rail transit professional documents. When a user performs a search, the system also vectorizes the user's query content and then performs similarity and keyword searches in the Elasticsearch database, returning the most relevant text blocks and their associated images, tables, and other information.

[0114] In addition to knowledge graphs, the knowledge base of the urban rail transit industry can also include: text summaries, table slices and / or multiple text vectors. By constructing a multi-dimensional knowledge base, multi-dimensional knowledge retrieval can be achieved, providing high-precision knowledge support for large-scale question answering and intelligent retrieval.

[0115] The knowledge base construction method for the RAG system provided in the above embodiments constructs a knowledge base that includes knowledge graphs, text summaries, table slice content, and / or multiple text vectors, so that information can be obtained from multiple dimensions when performing knowledge question answering or retrieval in the RAG system, thereby improving the effectiveness of the RAG system.

[0116] Based on the above method embodiments, this application provides a knowledge base construction apparatus for a RAG system. Figure 7 This is a schematic diagram of the structure of the knowledge graph construction device provided in the embodiments of this application, such as... Figure 7 As shown, the device may include: The document parsing module 601 is used to parse the original knowledge document according to the format of the original knowledge document in the urban rail transit industry and generate lightweight markup language text. The knowledge tree construction module 602 is used to split the lightweight markup language text according to the multi-level chapter titles and text content to generate a knowledge tree. Each leaf node of the knowledge tree is a piece of text content. The block generation module 603 is used to combine and segment multiple text contents under each level of heading according to a preset slicing threshold and a preset parent text block threshold to generate multiple blocks. The multiple blocks include: a first number of text blocks and a second number of parent text blocks. Each parent text block is a combination of at least two text blocks. Each block has the chapter titles of each level to which the target segment text content belongs. The knowledge graph construction module 604 is used to construct a knowledge graph of the original knowledge document by using a preset urban rail transit professional dictionary to perform entity recognition and relation extraction on multiple blocks. The knowledge base of the urban rail transit industry includes the knowledge graph, which is used to enable the RAG system to perform knowledge retrieval.

[0117] Optionally, the block generation module 703 is specifically used to combine and segment multiple text content segments according to their lengths and preset slicing thresholds to generate a first number of text blocks; generate a parent text block based on a segment of text content that has been segmented into at least two text blocks; sequentially combine a parent text block with other segments of text content under each level of heading, and update the parent text block according to the combined text length and the preset parent text block threshold to obtain a second number of parent text blocks; determine the level of chapter headings to which the target segment text content belongs from the knowledge tree based on the target segment text content corresponding to each text block or each parent text block; add corresponding level of chapter headings to each text block and each parent text block to generate each block.

[0118] Optionally, the block generation module 603 is further configured to: if the length of the first text segment is less than a preset slicing threshold, and the length of the first text segment combined with the second text segment is also less than the preset slicing threshold, continue to combine with other text segments until the length of the last combination is greater than or equal to the preset slicing threshold; generate a text block based on the text content of the previous combination; if the length of the first text segment is less than the preset slicing threshold, and the length of the first text segment combined with the second text segment is greater than or equal to the preset slicing threshold, determine that the first text segment generates a text block, and perform a combination judgment on the second and third text segments; if the length of the first text segment is greater than or equal to the preset slicing threshold, segment the first text segment to generate at least two text blocks.

[0119] Optionally, the block generation module 603 is also used to sequentially combine a parent text block with other paragraph text content under each level heading. After each combination of text content, it is determined whether the length of the text after each combination is greater than or equal to a preset parent text block threshold. If the length of the text after each combination is less than the preset parent text block threshold, a parent text block is updated to the length of the text after each combination. If the length of the text after each combination is greater than or equal to a preset slicing threshold, a parent text block is determined to remain unchanged.

[0120] Optionally, the knowledge graph construction module 604 is specifically used to perform dependency analysis on the statements in each block to determine the syntactic relations of the statements in each block; extract entities from each block based on the entity information in the urban rail transit professional dictionary to determine multiple entities in each block; extract relations from each block based on the relation information in the urban rail transit professional dictionary, the syntactic relations of the statements in each block, and the multiple entities in each block to determine the entity relations of the multiple entities in each block; determine the attribute information of the multiple entities in each block based on the attribute information in the urban rail transit professional dictionary and the multiple entities in each block; and generate a knowledge graph based on the entity relations of the multiple entities in each block and the attribute information of the multiple entities in each block.

[0121] Optionally, the document parsing module 601 is specifically used to: if the original knowledge document is in PDF format, perform layout detection on the original knowledge document to determine the content of the first table, the content of the first text block, and / or the content of the first image in the original knowledge document; recognize the content of the first table and perform hypertext markup language format conversion to generate a first JSON table; recognize the text and formulas of the first text block content respectively to generate the first text content and the formulas with a preset layout; store the content of the first image and generate a first index identifier for the storage file of the image content; and generate lightweight markup language text based on the first JSON table, the first text content, the formulas with a preset layout, and / or the first index identifier.

[0122] Optionally, the document parsing module 601 is specifically used to parse the original knowledge document if the original knowledge document is in docx format, determine the content of the second table, the content of the second text block, and / or the content of the second image in the original knowledge document; split the content of the second table according to preset rules to generate a second JSON table; obtain the second text content in the second text block content; store the second image content and generate a second index identifier for the storage file of the second image content; and generate lightweight markup language text based on the second JSON table, the second text content, and / or the second index identifier.

[0123] Optionally, the device further includes: The information generation module is used to generate text summaries based on the text content under the preset hierarchical chapter titles; and / or to generate table slice content based on the table summary, table title, and the chapter title to which the table belongs; and / or to perform text vectorization processing on multiple blocks to generate multiple text vectors; the knowledge base of the urban rail transit industry also includes: the text summaries, the table slice content, and / or the multiple text vectors.

[0124] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0125] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0126] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device 700 may include a processor 701, a storage medium 702, and a bus. The storage medium 702 stores program instructions executable by the processor. When the electronic device 700 is running, the processor 701 communicates with the storage medium 702 via the bus, and the processor 701 executes the program instructions to perform the method embodiment described above. Specific implementations and technical effects are similar and will not be repeated here.

[0127] Optionally, this application also provides a computer-readable storage medium storing a computer program, which is executed by a processor to perform the above-described method embodiments.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0129] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0131] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for constructing a knowledge base of a RAG system, characterized by, The method comprises: According to the format of the original knowledge document of the urban rail transit industry, the original knowledge document is parsed to generate a lightweight markup language text; According to the multi-level chapter title and the text content, the lightweight markup language text is split to generate a knowledge tree, and each leaf node of the knowledge tree is a piece of text content; According to the preset slicing threshold and the preset parent text block threshold, the multiple pieces of text content under each level of title are combined and divided to generate multiple blocks, and the multiple blocks include a first number of text blocks and a second number of parent text blocks, each parent text block is a combination of at least two text blocks, and each block has the chapter title of each level to which the target text content belongs; Using a preset urban rail transit professional dictionary, entity recognition and relationship extraction are performed on the multiple blocks to construct a knowledge graph of the original knowledge document, and the knowledge base of the urban rail transit industry includes the knowledge graph, and the knowledge base of the urban rail transit industry is used to enable the RAG system to perform knowledge retrieval.

2. The method of claim 1, wherein, According to the preset slicing threshold and the preset parent text block threshold, the multiple pieces of text content under each level of title are combined and divided to generate multiple blocks, including: According to the length of the multiple pieces of text content and the preset slicing threshold, the multiple pieces of text content are combined and divided to generate the first number of text blocks; According to the one piece of text content divided into at least two text blocks, one parent text block is generated; The one parent text block and the other pieces of text content under each level of title are combined in turn, the one parent text block is updated according to the length of the combined text and the preset parent text block threshold, and the second number of parent text blocks is obtained; According to the target text content corresponding to each text block or each parent text block, the chapter title of each level to which the target text content belongs is determined from the knowledge tree; The chapter title of each level corresponding to each text block and each parent text block is added to generate each block.

3. The method of claim 2, wherein, According to the length of the multiple pieces of text content and the preset slicing threshold, the multiple pieces of text content are combined and divided to generate the first number of text blocks, including: If the length of the first piece of text content is less than the preset slicing threshold, and the length of the combination of the first piece of text content and the second piece of text content is also less than the preset slicing threshold, the combination with other pieces of text content is continued until the length after the last combination is greater than or equal to the preset slicing threshold; According to the text content after the last previous combination, one text block is generated; If the length of the first piece of text content is less than the preset slicing threshold, and the length of the combination of the first piece of text content and the second piece of text content is greater than or equal to the preset slicing threshold, it is determined that the first piece of text content generates one text block, and the combination of the second piece of text content and the third piece of text content is judged; If the length of the first piece of text content is greater than or equal to the preset slicing threshold, the first piece of text content is divided to generate at least two text blocks.

4. The method of claim 2, wherein, The one parent text block and other segment text contents under each level title are combined in sequence, and the one parent text block is updated according to the length of the combined text and the preset parent text block threshold, including: The one parent text block and other segment text contents under each level title are combined in sequence, and the length of the combined text is determined each time a segment text content is combined; If the length of the combined text each time is less than the preset parent text block threshold, the one parent text block is updated to the length of the combined text each time; If the length of the combined text each time is greater than or equal to the preset parent text block threshold, it is determined that the one parent text block remains unchanged.

5. The method of claim 1, wherein, The preset urban rail professional dictionary is used to perform entity recognition and relationship extraction on the plurality of blocks to construct a knowledge graph of the original knowledge document, including: Dependency analysis is performed on the sentences of each block to determine the grammatical relationship of the sentences of each block; According to the entity information in the urban rail professional dictionary, entity extraction is performed on each block to determine a plurality of entities of each block; According to the relationship information in the urban rail professional dictionary, the grammatical relationship of the sentences of each block, and the plurality of entities of each block, relationship extraction is performed on each block to determine the entity relationship of the plurality of entities of each block; According to the attribute information in the urban rail professional dictionary and the plurality of entities of each block, the attribute information of the plurality of entities of each block is determined; According to the entity relationship of the plurality of entities of each block and the attribute information of the plurality of entities of each block, the knowledge graph is generated.

6. The method of claim 1, wherein, The original knowledge document is parsed according to the format of the original knowledge document of the urban rail transit industry to generate a lightweight markup language text, including: If the format of the original knowledge document is a PDF format, layout detection is performed on the original knowledge document to determine first table content, first text block content, and / or first picture content in the original knowledge document; The first table content is identified and converted into a HyperText Markup Language format to generate a first JSON table; The text and formula of the first text block content are identified respectively to generate first text content and a preset layout formula; The first picture content is stored to generate a first index identifier of a storage file of the picture content; The first JSON table, the first text content, the preset layout formula, and / or the first index identifier are used to generate the lightweight markup language text.

7. The method of claim 1, wherein, The original knowledge document is parsed according to the format of the original knowledge document of the urban rail transit industry to generate a lightweight markup language text, including: If the format of the original knowledge document is a docx format, the original knowledge document is parsed to determine second table content, second text block content, and / or second picture content in the original knowledge document; The second table content is split using a preset rule to generate a second JSON table; obtaining second text content in the second text block content; storing the second picture content, and generating a second index identifier of a storage file of the second picture content; generating the lightweight markup language text according to the second JSON table, the second text content, and / or the second index identifier.

8. The method of claim 1, wherein, The method further comprises: generating a text summary according to the text content under the preset hierarchical chapter title; and / or, generating table slice content according to the table summary, the table title, and the chapter title to which the table belongs; and / or, text vectorization processing on the plurality of blocks to generate a plurality of text vectors; the knowledge base of the urban rail transit industry further comprises: the text summary, the table slice content, and / or the plurality of text vectors.

9. An electronic device, comprising: comprise: a processor, a storage medium, and a bus, the storage medium storing program instructions executable by the processor, when the electronic device is running, the processor and the storage medium communicate through the bus, the processor executes the program instructions to execute the steps of the knowledge base construction method of the RAG system according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is executed by the processor to execute the steps of the knowledge base construction method of the RAG system according to any one of claims 1 to 8.

Images