Knowledge base construction method and device, computer device and storage medium

Abstract

The application relates to a knowledge base construction method and device, computer equipment and a storage medium. The method comprises the following steps: extracting a directory structure of a target document to obtain a directory structure tree; splitting the target document based on the directory structure tree to obtain a plurality of knowledge blocks; when the target document does not have label data, generating label data corresponding to the target document based on a preset large language model and an abstract text of the target document; and constructing a target knowledge base based on the label data of at least one target document and the plurality of knowledge blocks. The target knowledge base comprises a document tree composed of the label data of the target document. The method can improve the retrieval efficiency and accuracy of the knowledge base.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge base, in particular to a knowledge base construction method and device, computer equipment and storage medium. BACKGROUND

[0002] With the development of knowledge base technology, document tag management and content structuring become key links. However, in the traditional technology, the storage mode of the knowledge base, such as parallel storage, with the increase of the number of knowledge blocks, the amount of calculation consumed for retrieving and matching the knowledge blocks correspondingly increases a lot. On this basis, the way of retrieving knowledge blocks through cosine similarity not only is low in efficiency, but also may cause the matching degree of the actual demand knowledge blocks to be reduced due to different language expressions, resulting in the situation that the knowledge block retrieval accuracy is low.

[0003] Therefore, there is still the problem of low efficiency and accuracy of knowledge block retrieval in the prior art. SUMMARY

[0004] Therefore, there is still the problem of low efficiency and accuracy of knowledge block retrieval in the prior art.

[0005] In a first aspect, the present application provides a knowledge base construction method, which comprises:

[0006] extracting a directory structure tree from a target document;

[0007] splitting the target document based on the directory structure tree to obtain a plurality of knowledge blocks;

[0008] when the target document does not exist label data, generating label data corresponding to the target document based on a preset large language model and an abstract text of the target document;

[0009] constructing a target knowledge base based on the label data of at least one target document and the plurality of knowledge blocks; the target knowledge base comprises a document tree composed of the label data of the target document.

[0010] In one embodiment, the extracting a directory structure tree from a target document comprises:

[0011] extracting a first-level title and a second-level title of the target document based on outline level information, style level information or text matching rules of the target document;

[0012] if the extraction fails, taking the file name of the target document as the first-level title;

[0013] obtaining the directory structure tree based on the primary title, or the primary title and the secondary title.

[0014] In one of the embodiments, the obtaining the directory structure tree based on the target document further includes:

[0015] converting the target document into a lightweight markup language text;

[0016] extracting initial title information and abstract text of the target document based on the lightweight markup language text;

[0017] generating the directory structure tree corresponding to the target document based on a preset large language model, the initial title information and the abstract text.

[0018] In one of the embodiments, the splitting the target document based on the directory structure tree to obtain a plurality of knowledge blocks further includes:

[0019] cutting the text data corresponding to each title in the directory structure tree based on a preset sliding window and a preset step length to obtain a plurality of knowledge blocks.

[0020] In one of the embodiments, the constructing the target knowledge base based on the label data of at least one of the target documents and the plurality of knowledge blocks further includes:

[0021] determining a plurality of target knowledge bases corresponding to a text generation request in response to the text generation request;

[0022] merging the label data of the plurality of target knowledge bases to obtain a dynamic knowledge base and dynamic label data corresponding to the dynamic knowledge base;

[0023] performing post-tree pruning on the dynamic label data to obtain pruned label data;

[0024] obtaining a knowledge block in the pruned label data matching the text generation request;

[0025] obtaining a text generation result corresponding to the text generation based on a preset large language model, the text generation request and the knowledge block.

[0026] In one of the embodiments, the performing post-tree pruning on the dynamic label data to obtain pruned label data further includes:

[0027] calculating the routing complexity and the retrieval complexity of any two nodes in the dynamic label data before and after merging;

[0028] determining a weighted loss of the dynamic label data based on the routing complexity and the retrieval complexity;

[0029] If the weighted loss after merging is less than the weighted loss before merging, the two nodes are merged.

[0030] In one embodiment, the acquiring the knowledge block in the pruned label data that matches the text generation request comprises:

[0031] Acquiring a node in the pruned label data that matches the text generation request;

[0032] Acquiring a knowledge block under the node that matches the text generation request;

[0033] If the knowledge block under the node that matches the text generation request does not meet a preset condition, a parent node of the node is selected, and a knowledge block under the parent node that matches the text generation request is acquired.

[0034] In a second aspect, the present application provides a knowledge base construction device, the device comprising:

[0035] A directory extraction module is configured to extract a directory structure of a target document to obtain a directory structure tree.

[0036] A document splitting module is configured to split the target document based on the directory structure tree to obtain a plurality of knowledge blocks.

[0037] A label generation module is configured to generate label data corresponding to the target document based on a preset large language model and an abstract text of the target document when the target document does not have label data.

[0038] A knowledge base construction module is configured to construct a target knowledge base based on label data of at least one target document and a plurality of knowledge blocks; the target knowledge base comprises a document tree composed of the label data of the target document.

[0039] In a third aspect, the present application provides a computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the method described above when executing the computer program.

[0040] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described above.

[0041] The knowledge base construction method, device, computer device and storage medium described above can effectively reduce the redundant calculation amount in the knowledge block retrieval process, improve the semantic matching precision between the labels and the content, and optimize the retrieval path through the joint architecture of the document tree and the label index, so as to achieve the technical effects of improving the retrieval efficiency and accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 An application environment diagram of the knowledge base construction method in an embodiment;

[0043] Figure 2 A flowchart of the knowledge base construction method in an embodiment;

[0044] Figure 3 A flowchart of the knowledge base construction method in another embodiment;

[0045] Figure 4 A full document tree diagram of the knowledge base in an embodiment;

[0046] Figure 5 A diagram of merging of full document trees of multiple knowledge bases in an embodiment;

[0047] Figure 6 A structural block diagram of the knowledge base construction device in an embodiment;

[0048] Figure 7 An internal structure diagram of the computer device in an embodiment. DETAILED DESCRIPTION

[0049] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0050] The knowledge base construction method provided by the embodiments of the present application can be applied to, for example, Figure 1The application environment is shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The terminal 102 extracts the directory structure of the target document based on the control instruction of the server 104, obtains a directory structure tree; based on the directory structure tree, the target document is split to obtain a plurality of knowledge blocks; when the target document does not exist label data, based on a preset large language model and the abstract text of the target document, the label data corresponding to the target document is generated; based on at least one label data of the target document and a plurality of knowledge blocks, a target knowledge base is constructed; the target knowledge base includes a document tree composed of label data of the target document. Among them, the terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones and tablet computers. The server 104 can be realized by an independent server or a server cluster composed of multiple servers.

[0051] In one embodiment, as Figure 2 shown, a knowledge base construction method is provided, and the method is applied to the terminal 102 in Figure 1 for example, the knowledge base construction method comprises:

[0052] Step S100, the directory structure of the target document is extracted to obtain a directory structure tree.

[0053] Among them, the directory structure tree can be a tree data structure describing the hierarchical relationship of the document content, and the directory structure tree includes nodes, which can correspond to chapters, subchapters or specific content modules in the document. For example, the directory structure tree can be constructed by parsing the title, section symbol and other format marks in the document, and can also be combined with the recursive traversal method to obtain the hierarchical relationship.

[0054] For example, the directory structure tree can include one or more generated hierarchical structures such as the navigation structure of a Word document, the bookmark hierarchy of a PDF file, and the title tag based on an HTML document. In a specific embodiment, the parent-child node relationship can be established layer by layer by identifying the title text and its layout features of different levels in the document, and the directory structure tree with logical nested structure is formed. By extracting the inherent organizational structure information of the document, the structured expression of the overall framework of the document can be realized, thereby supporting the subsequent hierarchical logic division and organization of the document content.

[0055] Step S200, based on the directory structure tree, the target document is split to obtain a plurality of knowledge blocks.

[0056] The knowledge block can be the smallest unit of information carrying a specific topic or function, and its content can include but is not limited to text paragraphs, tables, charts, code snippets, etc. It can be generated by identifying natural separators in the document according to the node hierarchical relationship of the directory structure tree, combined with regular expression matching or natural language processing technology. For example, the division of the knowledge block can be based on one or more of chapter titles, blank lines, specific format markers, or semantic boundary detection results. In one specific embodiment, when the directory structure tree contains nodes such as "1. Introduction" and "1.1 Research Background", the document content in the corresponding range can be extracted as an independent knowledge block, and its hierarchical attribution information is retained. By deconstructing the original document in both semantic and structural dimensions, clear boundary and explicit topic information units can be formed, providing consistent data input for subsequent label generation and knowledge organization.

[0057] Step S300, when the target document does not exist label data, based on the preset large language model and the summary text of the target document, the label data corresponding to the target document is generated.

[0058] The label data can be a keyword or a classification identifier used to describe the characteristics of a knowledge block or a document topic. For example, the label type in the label data can be pre-set or generated according to the summary text. For example, when the user uploads the target document, the label type of the target document is labeled or the label content is filled in, it can be considered that the target document has label data.

[0059] When the target document does not have corresponding label data, the summary text of the target document can be extracted as input, and a large language model is used for semantic understanding and concept. The large language model is used to extract and generate labels. In this embodiment, the preset large language model can include but is not limited to a pre-training model based on the Transformer architecture, such as the GPT series, the BERT series, etc.

[0060] In one specific embodiment, the summary text can be extracted from the target document and input into the large language model. The model identifies core concepts through context understanding and combines domain knowledge base for semantic normalization processing, and outputs a label set that conforms to the preset classification system. For example, for the summary text "Application of deep learning in medical image diagnosis", the system can generate one or more of "artificial intelligence", "medical image", "neural network", etc. By using the semantic modeling capability of the large language model, automatic labeling of the document topic is realized, reducing manual intervention while improving the semantic consistency between the labels and the content.

[0061] Further, in the process of calling the large language model, the type of the label data can also be defined, for example, the large language model can be instructed to label from the industry attribute, domain attribute and type attribute of the document, so that multi-dimensional label data can be obtained.

[0062] At step S400, the target knowledge base is constructed based on the label data of the at least one target document and the plurality of knowledge blocks.

[0063] The target knowledge base can be a structured information set for storing, organizing and retrieving knowledge blocks, and the target knowledge base includes a document tree composed of label data of the target document. For example, the construction can be achieved by integrating knowledge block content, label index and document tree structure. The label index can be established by means of inverted index or hash table, etc. to realize the mapping relationship between the label and the corresponding knowledge block. The document tree can be a tree-like classification system composed of label data, which is organized by hierarchical association between labels. In an exemplary embodiment, "natural language processing" can be set as a parent node and "word vector model" as a child node. Further, in a specific embodiment, a graph database technology (such as Neo4j) can also be used to store label nodes and their mutual relationships to support complex path queries and dynamic expansion. Through multi-dimensional data integration, a knowledge management system with semantic correlation and structural navigability can be formed.

[0064] The knowledge base construction method provided in the embodiment extracts the directory structure of the target document to obtain a directory structure tree, splits the target document based on the directory structure tree to obtain a plurality of knowledge blocks, generates corresponding label data using a preset large language model in combination with the document summary text in the case where the target document lacks label data, and completes the construction of the target knowledge base containing a document tree composed of label data by integrating the label data and the knowledge blocks. Through the synergistic effect of structured deconstruction, semantically driven label generation and multi-modal organization mechanism, the redundant calculation amount in the knowledge block retrieval process can be effectively reduced, the semantic matching precision between the label and the content can be improved, and the retrieval path can be optimized through the joint architecture of the document tree and the label index, so that the technical effects of improving the retrieval efficiency and accuracy can be achieved.

[0065] In one embodiment, the directory structure of the target document is extracted to obtain a directory structure tree, which includes:

[0066] Based on the outline level information, style level information or text matching rule of the target document, the first-level title and the second-level title of the target document are extracted;

[0067] If the extraction fails, the file name of the target document is used as the first-level title;

[0068] Based on the first-level title, or the first-level title and the second-level title, the directory structure tree is obtained.

[0069] The outline level information can be a structured markup in the target document for identifying the paragraph level, which can be obtained by parsing the title level attribute in the document object model. For example, the outline level information can include one or more of the style tags such as "Heading 1", "Heading 2", etc., which can reflect the explicit division of the document logical structure by the document editor during the editing process.

[0070] The style level information can be a formatting feature in the document paragraph that reflects the visual level difference, which can be determined by analyzing one or more of the layout attributes including but not limited to font size, bold state, paragraph indentation, line spacing, etc. For example, the style level information can also include one or more of the text with font size significantly larger than the body text, the paragraph with bold centered, the line with specific indentation, etc., which can be used to infer the potential title in the document without using the standard outline markup.

[0071] The text matching rule can be a text analysis strategy for identifying the title semantics or format pattern, which can be performed by regular expression, keyword matching, position feature recognition, etc. For example, the text matching rule can include one or more of matching the numbered format such as "Chapter X", "X.X", identifying the fixed chapter name such as "Abstract", "Introduction", "References", detecting the text block located at the top of the page with larger font, etc., so that the implicit structure information can be extracted from the document without explicit style or outline markup.

[0072] The primary title can be the highest level chapter identifier in the document, and the secondary title can be the sub-chapter identifier belonging to the primary title, both of which constitute a hierarchical structure unit with parent-child relationship.

[0073] Extracting the primary title and the secondary title can be one or more of the outline level information, the style level information, and the text matching rule applied alternatively or in parallel for title extraction. For example, applying the outline level information, the style level information, and the text matching rule in parallel can be to fuse and deduplicate the multi-source results, so that the title content with hierarchical relationship can be identified in the document with different format specification degrees, and the integrity and accuracy of title extraction are improved.

[0074] The file name can be the naming identifier of the target document in the storage system, which can be obtained by reading the filename field in the document metadata. When it is determined that the foregoing title extraction process does not output the valid title node, and when the determination fails, the default title generation mechanism is triggered, and the file name is taken as the primary title, so that even in the extreme case of completely lacking structured or formatted information, the directory structure containing at least the top node can still be generated, and the interruption of the subsequent processing process due to the lack of structure is avoided.

[0075] Based on the first-level title, or the first-level title and the second-level title, a directory structure tree is obtained, which can be an exemplary tree data structure constructed with the first-level title as the root node, and the second-level title as the child node mounted under the corresponding first-level title to form a multi-level directory structure. If there is only a first-level title or only a single title generated by a file name, a single-layer tree structure containing only a root node is constructed. In a specific embodiment, if the document successfully extracts "Chapter 1 Introduction" as a first-level title and "1.1 Research Background" as a second-level title, the directory structure tree can contain two-level nodes; if another document can only generate a title by the file name "New Energy Technology White Paper", the directory structure tree contains only one first-level title as the top node. By dynamically adjusting the depth and branch structure of the tree according to the actual extraction result, the hierarchical logic of the original document can be preserved, while the document form with simplified structure can be adapted.

[0076] The knowledge base construction method provided in this embodiment extracts the directory structure of the target document to obtain a directory structure tree, including extracting a first-level title and a second-level title based on the outline level information, style level information or text matching rules of the target document, if the extraction fails, the file name is taken as a first-level title, and a directory structure tree of the corresponding level is constructed based on the extraction result, multi-dimensional title recognition is realized by fusing the structured mark of the document, visual format features and text semantic patterns, the continuity of the extraction process is guaranteed by combining the file name bottom-up strategy, so that stable and reasonable directory structure can be generated when processing standard and non-standard format documents, especially when facing documents without style marks or with chaotic formats, it still has strong adaptability, significantly improves the coverage and reliability of structure analysis, and then improves the accuracy of subsequent content splitting, avoids knowledge organization deviation caused by structure loss or misjudgment, and can achieve the technical effect of improving the robustness of document structured processing.

[0077] In one of the embodiments, the directory structure extraction of the target document to obtain the directory structure tree further includes:

[0078] Converting the target document into a lightweight markup language text;

[0079] Extracting initial title information and abstract text of the target document based on the lightweight markup language text;

[0080] Generating a directory structure tree corresponding to the target document based on the preset large language model, the initial title information and the abstract text.

[0081] The lightweight markup language text can be a structured text expressed in a lightweight markup language format, which labels the hierarchical structure and content elements of the document through specific syntax symbols. For example, the original document, including but not limited to one or more of Word documents, PDF files, etc., can be converted into a lightweight markup language text by a parsing tool. In one specific embodiment, the document format conversion can be to convert the target document of different sources into a lightweight markup language text by using an existing lightweight markup language library or parsing tool, so as to realize the standardization of the document representation form, eliminate the processing compatibility problems caused by the format difference of the document, and at the same time, retain the title level relationship and the text content in the original document.

[0082] The initial title information can be the title content at each level and its nested structure in the document according to the lightweight markup language syntax. For example, the line starting with a symbol such as “#” can be matched by a regular expression or recognized based on a parsing library to extract the first-level, second-level, etc. title text and its level relationship.

[0083] The abstract text can be an unstructured text paragraph reflecting the summary of the core content of the document. For example, the continuous paragraph at the beginning of the document or the area containing keywords such as “abstract” and “introduction” can be extracted. In one specific embodiment, the separation and extraction process of the title and the abstract can be performed based on the converted lightweight markup language text, so as to obtain the explicit structure framework and key semantic information of the document.

[0084] The preset large language model can be a text generation model based on the Transformer architecture. For example, it can be one or more of GPT-4, Tongyi Qianwen, etc. The large language model receives the initial title information and the abstract text as input, analyzes the context semantics of the title sequence and understands the theme distribution of the abstract content, and outputs a directory structure tree with logical coherence. In one specific embodiment, when the initial title has a hierarchical order anomaly (such as a second-level title appearing before a first-level title), the model can automatically adjust the node ownership according to the context semantics; if the abstract mentions that a chapter covers multiple sub-topics, the model can infer and suggest adding a third-level title node. Further, a prompt word engineering can be used to make the large language model output a text in a specified format, so as to convert the format text into, for example, a dictionary object through a type conversion tool, so as to improve the parsing efficiency. By reconstructing the document structure in a semantic-driven manner, the limitations of traditional extraction methods that only rely on explicit markers are broken through, and the structure restoration capability of complex or non-standard documents is improved.

[0085] The knowledge base construction method provided in the embodiment can realize high-precision restoration of the actual content structure of a document, so as to achieve the effect of generating a more accurate and semantically coherent directory structure tree.

[0086] In one of the embodiments, based on the directory structure tree, the target document is split to obtain a plurality of knowledge blocks, including:

[0087] Based on a preset sliding window and a preset step, the text data corresponding to each title in the directory structure tree is sequentially divided to obtain a plurality of knowledge blocks.

[0088] The preset sliding window can be a data processing mechanism for controlling the text division granularity, and the width of the window can be the length of the text segment contained in a single knowledge block. The length of the text segment can be measured by the number of characters, the number of sentences, the number of paragraphs, etc. The window length can be set through a configuration file, user input or system default value. In one exemplary embodiment, when processing technical documents, the text amount of 200 characters or two paragraphs can be set.

[0089] The preset step can be the length of the text spanned by the sliding window each time it moves on the text sequence, and its value is less than or equal to the window length, which is used to control the degree of overlap between adjacent knowledge blocks. The step can be configured by a system parameter or adaptively adjusted according to the document type. When the step is less than the window length, there will be an overlapping area between adjacent knowledge blocks, thereby preserving the semantic continuity of the context.

[0090] The text data is divided based on a preset sliding window and a preset step size. For example, the original text area covered by each title can be determined according to the directory structure tree. The preset window length and step size parameters are read, and the parameter values can be adjusted according to the ideal information density of the knowledge block or the application scenario requirement. The text content is iterated, and the window is slid step by step, and the text segment contained in the window is encapsulated as an independent knowledge block. By adjusting the window length and step size, the information granularity of the knowledge block can be flexibly controlled, and the problem of redundant knowledge block content caused by too long chapters can be avoided. The overlapping area can make the key information across the title boundary repeatedly contained in multiple knowledge blocks, and the fault tolerance of information retrieval is improved. Further, according to the document type, a corresponding sliding window width and a preset compensation can be selected. For example, for legal documents, the step size can be set equal to the paragraph length to ensure accurate reference, and for papers, a smaller step size can be used to generate overlapping blocks to capture the continuity of the argument logic.

[0091] The knowledge base construction method provided by the embodiment includes: based on a directory structure tree, a target document is split to obtain a plurality of knowledge blocks, including: based on a preset sliding window and a preset step size, the text data corresponding to each title under the directory structure tree is sequentially divided to obtain a plurality of knowledge blocks. The document hierarchical structure and the parameterized sliding window mechanism are combined to realize fine-grained division of text content, the window length is used to control the information capacity of the knowledge block, and the step size is used to adjust the overlap degree of adjacent knowledge blocks to retain semantic continuity, thereby improving the semantic coherence and retrieval adaptability of the knowledge block as a whole, reducing the information fragmentation phenomenon caused by unreasonable structure division, and achieving the technical effects of enhancing the context perception ability of the model and improving the generation accuracy of the model.

[0092] In one of the embodiments, after constructing the target knowledge base based on the label data of at least one target document and the plurality of knowledge blocks, the method further includes:

[0093] In response to a text generation request, a plurality of target knowledge bases corresponding to the text generation request are determined;

[0094] The label data of the plurality of target knowledge bases is merged to obtain a dynamic knowledge base and dynamic label data corresponding to the dynamic knowledge base;

[0095] The dynamic label data is pruned after the tree to obtain pruned label data;

[0096] The knowledge block matching the text generation request in the pruned label data is obtained;

[0097] Based on a preset large language model, a text generation request, and a knowledge block, a text generation result corresponding to the text generation is obtained.

[0098] The text generation request can be a natural language instruction or a task description input by a user to express a demand for specific content generation, and can be obtained through receiving a front-end interface input, API calling, or voice recognition conversion.

[0099] The multiple target knowledge bases can be multiple knowledge collections storing structured knowledge blocks and their label data, used to support information retrieval and content generation in different fields, different topics, or different types. Each target knowledge base can be constructed using the knowledge base construction method in the above embodiments, which will not be repeated here.

[0100] Determining the multiple target knowledge bases corresponding to the text generation request can be performed by performing semantic analysis on the text generation request, extracting keywords, identifying intent, and comparing the semantic matching or vector similarity calculation with the topic labels of each knowledge base, so as to filter out a knowledge base set with high relevance, and dynamically associate multiple knowledge sources according to the request content.

[0101] The dynamic knowledge base can be a knowledge organization structure temporarily constructed in response to a specific text generation request, which can include knowledge blocks and integrated label systems from multiple target knowledge bases, and form a cross-domain composite knowledge network through merging operations. The dynamic label data can be a unified label system used to organize and index the knowledge blocks in the dynamic knowledge base, which can be generated through label normalization, hierarchical reconstruction, and semantic fusion.

[0102] It can be understood that the label data of the multiple target knowledge bases can have differences in structure and terminology. Merging the label data of the multiple target knowledge bases can include measures such as term standardization, semantic alignment, and redundancy elimination. For example, a unified label space can be constructed to map the original labels of each knowledge base to the space, and the labels can be merged and reorganized according to semantic equivalence or hierarchical relationship, so as to form a dynamic knowledge organization structure that covers multi-domain knowledge and has consistent structure.

[0103] The pruned label data can be a subset of labels highly relevant to the current request after filtering, which can be obtained by removing low-relevance or irrelevant branches. Post-pruning the dynamic label data, for example, the semantic similarity between each label node and the text generation request can be calculated, and a threshold can be set to filter low-score nodes, or an information gain-based algorithm can be used to evaluate the contribution of nodes and prune redundant branches, so as to reduce the knowledge retrieval range and improve the subsequent matching efficiency.

[0104] The knowledge block matched with the text generation request can be obtained through a label guiding and content semantic double matching mechanism. For example, a hybrid retrieval strategy can be used to screen the optimal candidate knowledge block by combining the label path weight with the semantic similarity score between the content of the knowledge block and the request text.

[0105] Based on the preset large language model, the text generation request, and the knowledge block, the text generation result corresponding to the text generation can be obtained by using a prompt word engineering method to splice the text generation request and the knowledge block into an input prompt and inputting the input prompt into the large language model for inference generation, so that the model can refer to the real knowledge content during the generation process.

[0106] The knowledge base construction method provided in this embodiment can complete cross-domain integration of the label system, optimize the retrieval range by using label tree pruning, and enhance the factual reliability of the generated content by injecting structured knowledge into the large language model, thereby achieving the technical effects of reducing the knowledge retrieval computing overhead, improving the field relevance and professional accuracy of the generated content when processing text generation tasks in cross-domain and complex topics.

[0107] In one of the embodiments, the post-tree pruning of the dynamic label data includes:

[0108] The routing complexity and the retrieval complexity of any two nodes in the dynamic label data before and after merging are calculated.

[0109] Based on the routing complexity and the retrieval complexity, the weighted loss of the dynamic label data is determined.

[0110] If the weighted loss of the two nodes after merging is less than the weighted loss before merging, the two nodes are merged.

[0111] The node can be a specific knowledge classification or keyword in the label tree, which is used to organize and represent the information unit in the knowledge base.

[0112] The routing complexity can be an index reflecting the hierarchical depth or the number of jumps required from the root node to the target node in the label tree, which is used to measure the structural complexity of the path and can be obtained by traversing and analyzing the hierarchical position of the node in the tree structure. For example, the routing complexity can be calculated based on the depth value of the node, the path length of the parent node, and other parameters, including but not limited to using a tree traversal algorithm to obtain the path level number.

[0113] The retrieval complexity can be an index reflecting the number of operations required to find its associated knowledge block through a certain node, and is related to the number of child nodes under the node and the distribution of knowledge blocks. The retrieval complexity is used to evaluate the efficiency of information retrieval, and can be determined by counting the number of knowledge entries under the node, the number of branch judgment times, etc. For example, the retrieval complexity can include one or more of the knowledge access cost calculation methods based on the number of child nodes.

[0114] The weighted loss can be an evaluation function that combines the routing complexity and the retrieval complexity, and is used to represent the overall operation cost of the label tree structure. It can be generated by assigning different weights to the two types of complexity and linearly combining them. The weights of the two types of complexity can be determined by a configuration file, user input, or a system adaptive adjustment mechanism. For example, in a scenario where retrieval response speed is prioritized, the weight of the retrieval complexity can be set to a value higher than that of the routing complexity, to enhance the influence of the retrieval complexity in the total loss.

[0115] Node merging can be an adjustment of the hierarchical relationship between two nodes in the label tree, integrating them into one node or uniformly belonging to a newly generated parent node, to simplify the label structure and reduce redundant branches. For example, the numerical value of the weighted loss before and after merging can be compared, and when the weighted loss after merging is lower than that before merging, the update operation of the node structure is triggered. In one exemplary embodiment, if the merging results in a decrease in routing complexity due to a reduction in hierarchy, and a decrease in retrieval complexity due to a reduction in branch judgment, the overall weighted loss decreases, meeting the merging condition, and the system performs structure merging; otherwise, the original structure is retained.

[0116] The knowledge base construction method provided in this embodiment calculates the routing complexity and retrieval complexity of any two nodes in the dynamic label data before and after merging, determines the weighted loss of the dynamic label data based on the routing complexity and the retrieval complexity, and merges the two nodes if the weighted loss after merging is less than the weighted loss before merging. The routing complexity and the retrieval complexity are used to quantitatively model the label tree structure, and the overall efficiency change brought by node merging is evaluated in combination with the weighted loss function. The node merging decision is driven based on the comparison result of the loss value, which reduces the retrieval operation cost while reducing the hierarchical depth of the label tree, avoids structural imbalance caused by excessive or insufficient pruning, and can achieve the technical effect of improving the compactness of the label structure and the cooperative optimization of retrieval efficiency.

[0117] In one of the embodiments, obtaining the knowledge block in the pruned label data that matches the text generation request includes:

[0118] Obtaining the node in the pruned label data that matches the text generation request.

[0119] acquiring knowledge blocks under the matched node that match the text generation request;

[0120] If the knowledge blocks under the matched node that match the text generation request do not meet the preset condition, a parent node of the node is selected, and knowledge blocks under the parent node that match the text generation request are acquired.

[0121] The acquiring of the node that matches the text generation request in the pruned label data can be performed by calculating semantic similarity between the text generation request and each label node and screening nodes with a similarity higher than a preset threshold.

[0122] The preset condition can be an evaluation criterion set for the quality or quantity of the knowledge block set, for judging whether the current hierarchical retrieval result is sufficient to support high-quality text generation. Exemplarily, the preset condition can include, but is not limited to, one or more of a knowledge block quantity threshold (such as ≥ 3), a time validity requirement (such as a proportion of data in the last five years ≥ 60%), a content relevance score (such as an average semantic similarity ≥ 0.7), and an expert annotation credibility.

[0123] The selection of the parent node of the node can be a traversal of the hierarchical structure of the label tree, a one-level backtracking, and a verification of a path in which the parent node exists. The acquiring of the knowledge blocks under the parent node that match the text generation request can be a re-execution of the knowledge block retrieval process at the parent node level and a screening of the knowledge blocks that meet the condition in combination with the semantic similarity.

[0124] It can be understood that when the knowledge blocks retrieved by the parent node still do not meet the preset condition, the one-level backtracking can be continued until the root node is reached, and the final knowledge blocks are obtained.

[0125] The knowledge base construction method provided in this embodiment can acquire a node that matches a text generation request in pruned label data, acquire knowledge blocks under the matched node that match the text generation request, select a parent node of the node if the knowledge blocks under the matched node that match the text generation request do not meet a preset condition, and acquire knowledge blocks under the parent node that match the text generation request. Through the hierarchical matching and backtracking mechanism based on the hierarchical structure of the label tree, when the knowledge blocks in the initially matched node do not meet the evaluation criteria such as quantity, timeliness, or relevance, the one-level backtracking is automatically performed to the parent node with a wider semantic range to expand the retrieval range, and efficient retrieval of multi-level knowledge blocks is implemented, thereby achieving the technical effects of improving the success rate of knowledge block matching and enhancing the completeness and reliability of the text generation content.

[0126] In order to more clearly set forth the technical solutions of the present application, a detailed embodiment is further provided.

[0127] In one embodiment, as shown in Figure 3 a knowledge base construction method is provided, including:

[0128] I. Knowledge Base Construction.

[0129] (a) Construction of document summary and directory structure tree.

[0130] 1. Abstract Extraction: Based on keyword targeting, such as "abstract" and "conclusion". If no corresponding abstract is obtained by keyword targeting, the document is segmented by paragraph, and each document is iteratively summarized until all paragraphs are summarized, at which point a summary for that single document is generated.

[0131] 2. Directory Structure Tree Construction: A directory structure tree, also known as a directory tree, refers to the storage of document content into knowledge blocks according to a title hierarchy. Each knowledge block has corresponding title metadata, which records the title level of the knowledge block within the document. For documents without a title hierarchy, a single-node directory structure tree is constructed based on the document name. This embodiment involves processing Word and PDF documents. The following processing methods can also be applied to the recognition of other document formats with adaptive adjustments, which will not be elaborated upon here.

[0132] Word's heading hierarchy recognition includes:

[0133] Step 1: Read the outline level information from the Word document and retrieve the first-level, second-level, and other heading information based on this information. If the outline level information can be read, construct a table of contents structure tree. If the outline level information cannot be read, proceed to Step 2.

[0134] Step 2: Read the style level information of the Word document and retrieve the first-level, second-level, and other heading information based on this information. If the style level information can be read, construct a directory structure tree. If the style level information cannot be read, proceed to Step 3.

[0135] Step 3: Extract the heading hierarchy based on custom rules. Only extract the first-level headings and second-level headings. As long as any one of the rules is met, the current paragraph is a first (second) level heading.

[0136] In one specific embodiment, a custom first-level heading rule may include:

[0137] (1) The current paragraph text information begins with: Chapter XX, Section XX, Part XX, etc.

[0138] (2) The current paragraph text information begins with a number and “.”; such as “1.”, “2.”, “3.”.

[0139] (3) The current paragraph text information only includes background, summary or conclusion.

[0140] (4) Current paragraph text information: font style is centered, and text length is within 20 characters.

[0141] In one specific embodiment, the custom secondary title rule can include:

[0142] (1) The current paragraph text information starts with: Chapter XX, Section XX, Part XX, etc.

[0143] (2) The current paragraph text information starts with: numbers and "." and contains secondary numbering (e.g. 1.1, 1.2, 2.1, 2.2, etc.);

[0144] Step 4: If the first three steps cannot extract the title structure of the word document, the default word file name is used as the first-level title, i.e. the current document tree structure only has a first-level title structure.

[0145] The title hierarchy structure recognition of the pdf includes:

[0146] Step 1: Use existing pdf-to-markdown technology to obtain the markdown content of the pdf document.

[0147] Step 2: Based on the markdown obtained from the pdf document, the directory structure tree is obtained. Since the result of pdf-to-markdown has some noise, there is a possibility of title hierarchy recognition error. This embodiment provides a directory structure extraction method based on a large model, including:

[0148] Based on the extracted markdown file, the extracted different level title information is obtained. The large model prompt word is constructed, and the exemplary prompt word can be: "You are a document directory generation expert, combined with the summary of the current document: (summary text), and the preliminary title information: (extracted different level title information), correctly extract the directory structure of the document from the semantic logical level, label level, etc. Information angle, the result is returned in json format".

[0149] Step 3: Based on the json result returned by the large model, the directory structure tree of the document is parsed.

[0150] 3. Knowledge segmentation based on directory structure.

[0151] After obtaining the directory structure tree, the text data under each title is segmented into knowledge blocks according to the sliding window method: a fixed width window is used to move along the document, and a certain step size is moved each time. Each knowledge block is configured with corresponding title label source data.

[0152] (II) Single knowledge base sub-document tree construction.

[0153] In this embodiment, the document management method of the knowledge base is to label each document with corresponding labels, and documents with the same label are stored in the same location, that is, all documents in the same knowledge base in this embodiment will be assigned corresponding labels, and the documents in the knowledge base are stored and managed using a tree data structure according to the label data.

[0154] The document label generation method includes:

[0155] 1. Manual labeling. The label data of three layers of attributes can be preset from industry attributes, field attributes, and type attributes, such as Figure 4 As shown, the attributes of the documents are configured in the industry attributes, field attributes, and type attributes, and the user can configure the attribute labels of the documents when uploading the documents to the knowledge base.

[0156] 2. Automatic labeling. If it is detected that the user has not configured any label data for the document when uploading, the summary obtained based on the pre-step "summary extraction" can be used to construct a model system prompt word based on the large model capability. In a specific embodiment, the prompt word can be: "You are a document management master. You will define the document from multiple dimensions such as industry attributes, field attributes, and type attributes based on the summary of the document: [fill in the summary content of the document], and output the specific label values of different attributes. The result is returned in json format. Please note that you do not need to be limited to the above three attributes. Please perform semantic analysis and sentiment analysis based on the specific content of the summary to obtain the final result", so that the corresponding label data can be automatically labeled on the document.

[0157] (Three) Construction of a full document tree based on multiple knowledge bases.

[0158] When the user specifies to retrieve an answer from multiple knowledge bases for the current question, a full document tree can be merged from the specified multiple knowledge base sub-trees, and tree pruning operations can be performed, including:

[0159] 1. Attribute merging.

[0160] The attribute merging logic of the full document tree includes: the attribute values of each layer of the tree are taken as a union set, such as the attribute values of the first layer of the sub-document tree i T i，l ={f1, f2, f3}, the attribute values of the first layer of the sub-document tree j T j，l ={f4, f5}, then the first layer of the node attribute of the merged full document tree T u，l i and j is T u，l =T i，l ∪T j，l ={f1, f2, f3, f4}. As shown in Figure 5 The attribute merging of knowledge base A and knowledge base B can be a union set of knowledge base A and knowledge base B.

[0161] 2. Post-pruning the whole tree.

[0162] When the number of knowledge bases, the number of documents, and the number of classification attributes are more, the number of leaf nodes of the merged whole document tree is more, which requires higher ability of the large model to route to the accurate leaf node according to the user question. In order to reduce the routing ability of the large model, it is necessary to reduce the complexity of the whole document tree of the knowledge base, but reducing the complexity of the whole document tree will increase the complexity of the knowledge block retrieval space. Therefore, the embodiment proposes a knowledge base tree post-pruning method based on dynamic balance of routing complexity and retrieval complexity, and the pruning process includes:

[0163] (1) Calculate the routing complexity. The routing complexity is to measure the accurate routing ability of the large model, that is, the ability of the large model to predict the attribute value of the question at each layer of the whole document tree.

[0164] First step: according to the directory structure of each document and its corresponding content, use the text generation ability of the large model to randomly generate a number of questions. Then use the attributes of each level of the current whole document tree to construct the large model system prompt word. For example, the prompt word can be "You are a question searching master, there is a three-layer document tree management structure, the first layer has: traffic, public safety, finance and education attributes; In the second layer: traffic police have signal control, safety analysis attributes, … Now the question is: What is the adaptive signal control calculation method at the intersection? Which leaf node in the above tree structure may the corresponding answer come from", each question is based on the prompt word to let the large model generate the attribute value of the question routed to the leaf node of the current document tree according to the predicted attribute value routed to the leaf node of the current document tree.

[0165] Second step: calculate the question routing loss function H of the current whole document tree t .

[0166]

[0167] Where N t is the number of samples contained in the leaf node t, N tk is the number of samples in the leaf node t that the question comes from the leaf node k. For the leaf node t, if all sample data belong to the same class, the node loss function is 0, indicating that the node is very accurate.

[0168] (2) Calculate the retrieval complexity. The retrieval complexity is to describe the number of documents retrieved by the leaf node. If the number of documents of the leaf node is more, the retrieval complexity is greater. It can be measured by the number of leaf nodes of the document tree. If the number of leaf nodes is more, then the average number of documents T of each leaf node is less, so the retrieval complexity N t =1 / T.

[0169] (3) Document tree loss function. The empirical loss function L of the document tree is the weighted loss of routing complexity and retrieval complexity by weight a:

[0170] L = H t + aN t

[0171] The loss functions before and after pruning are calculated respectively, and are L before , L after , if the loss function before pruning is greater than the loss function after pruning, the branch can be cut off, and a leaf node is synthesized.

[0172] II. Knowledge block retrieval and generation based on hierarchical structure.

[0173] (I) Knowledge retrieval and generation.

[0174] The user inputs a question, and the system will generate a label based on the prompt word Prompt q The large model question answer may come from what type of document. The large model generates a label I route Routes to a specific full document tree node to obtain all documents d a in the corresponding leaf node document library. Then knowledge retrieval is performed in the corresponding knowledge block d a , a hybrid retrieval engine mechanism of vector retrieval and inverted index is adopted, and a Cross-Encoder model reordering mechanism is adopted to obtain the topN knowledge blocks.

[0175] Constructing knowledge logic: merging the topN knowledge blocks recalled into the prompt word template Prompt g for answer generation, for example, the prompt word template can be "Retrieved the following content: {knowledge block i title information} from {knowledge block content information}, please answer: {user question} based on the above information, if the information is insufficient, please explain the reason", thereby obtaining the answer output.

[0176] (II) Abnormal scenario processing.

[0177] Due to the existence of many large model sentiment analysis, type identification and other capabilities in the middle process, if the large model encounters ambiguous questions, ambiguous documents and other scenarios, it may cause problem routing errors, full document tree construction errors and other situations, resulting in problems cannot be correctly retrieved and answers generated. Therefore, the following abnormal situation processing is proposed in this embodiment:

[0178] The number of topN knowledge blocks retrieved is 0, or the generated result is: insufficient information and reason scenario. Take the parent node of the current label routing node, take the document corresponding to the parent node, and retrace the retrieval according to the steps "knowledge retrieval and generation" until the root node position.

[0179] The knowledge base construction method provided by the embodiment can achieve the technical effects of improving retrieval efficiency and accuracy. The directory structure tree is obtained by extracting the directory structure of the target document. The first-level title and the second-level title are extracted based on the outline level information, the style level information or the text matching rule of the target document. If the extraction fails, the file name is used as the first-level title. The directory structure tree of the corresponding level is constructed based on the extraction result, which can improve the robustness of document structured processing. The target document is converted into lightweight markup language text. The initial title information and the abstract text of the target document are extracted based on the lightweight markup language text. The directory structure tree corresponding to the target document is generated based on the preset large language model, the initial title information and the abstract text. The format standardization processing eliminates the parsing differences of multiple source documents. The structured title and the unstructured abstract are combined to enhance the understanding ability of the model to the document logic. The semantic analysis mechanism is used to identify the implicit chapter association and the hierarchical error. The actual content structure of the document can be restored with high precision, so that a more accurate and semantically coherent directory structure tree can be generated. The directory structure tree is used to split the target document to obtain multiple knowledge blocks. The text data corresponding to each title in the directory structure tree is cut based on the preset sliding window and the preset step length to obtain multiple knowledge blocks. The context perception ability of the model is enhanced, and the accuracy of the model is improved. In response to a text generation request, multiple target knowledge bases are determined. The label data of the multiple target knowledge bases is merged to form a dynamic knowledge base and corresponding dynamic label data. The dynamic label data is pruned to obtain pruned label data. The knowledge block matched with the text generation request is obtained. The text generation result is generated based on the preset large language model, the text generation request and the knowledge block. The technical effects of reducing the knowledge retrieval calculation overhead, improving the relevance and professional accuracy of the generated content in the processing of cross-domain and complex topic text generation tasks can be achieved. The routing complexity and the retrieval complexity of any two nodes in the dynamic label data before and after merging are calculated. The weighted loss of the dynamic label data is determined based on the routing complexity and the retrieval complexity. If the weighted loss of the two nodes after merging is less than the weighted loss before merging, the two nodes are merged. The technical effects of improving the compactness of the label structure and the collaborative optimization of the retrieval efficiency can be achieved.By acquiring the node matched with the text generation request in the pruning label data, acquiring the knowledge block matched with the text generation request under the node, if the knowledge block matched with the text generation request under the node does not satisfy the preset condition, selecting the parent node of the node, and acquiring the knowledge block matched with the text generation request under the parent node, the technical effects of improving the knowledge block matching success rate and enhancing the integrity and reliability of the text generation content can be achieved.

[0180] It should be understood that, although each step in the flowchart involved in each embodiment as described above is shown in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless explicitly stated herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0181] Based on the same inventive concept, the embodiments of the present application also provide a knowledge base construction device for implementing the above-mentioned knowledge base construction method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more knowledge base construction device embodiments provided below can refer to the limitations of the knowledge base construction method in the above text, which will not be repeated here.

[0182] In one embodiment, as shown in Figure 6 a knowledge base construction device is provided, and the device comprises:

[0183] The directory extraction module 100 is configured to extract the directory structure of the target document to obtain a directory structure tree.

[0184] The document splitting module 200 is configured to split the target document based on the directory structure tree to obtain a plurality of knowledge blocks.

[0185] The label generation module 300 is configured to generate label data corresponding to the target document based on a preset large language model and the abstract text of the target document when the target document does not have label data.

[0186] The knowledge base construction module 400 is configured to construct a target knowledge base based on the label data of at least one target document and the plurality of knowledge blocks. The target knowledge base comprises a document tree composed of the label data of the target document.

[0187] In one of the embodiments, the directory extraction module 100 is further configured to:

[0188] extract a first-level title and a second-level title of the target document based on the outline-level information, the style-level information, or the text matching rule of the target document;

[0189] if the extraction fails, take a file name of the target document as the first-level title;

[0190] obtain the directory structure tree based on the first-level title, or the first-level title and the second-level title.

[0191] In one of the embodiments, the directory extraction module 100 is further configured to:

[0192] convert the target document into a lightweight markup language text;

[0193] extract initial title information and abstract text of the target document based on the lightweight markup language text;

[0194] generate a directory structure tree corresponding to the target document based on a preset large language model, the initial title information, and the abstract text.

[0195] In one of the embodiments, the document splitting module 200 is further configured to:

[0196] split text data corresponding to each title in the directory structure tree in sequence based on a preset sliding window and a preset step length to obtain a plurality of knowledge blocks.

[0197] In one of the embodiments, the apparatus further comprises a text generation module configured to:

[0198] determine a plurality of target knowledge bases corresponding to a text generation request in response to the text generation request;

[0199] merge label data of the plurality of target knowledge bases to obtain a dynamic knowledge base and dynamic label data corresponding to the dynamic knowledge base;

[0200] perform post-tree pruning on the dynamic label data to obtain pruned label data;

[0201] obtain a knowledge block in the pruned label data that matches the text generation request;

[0202] obtain a text generation result corresponding to the text generation based on a preset large language model, the text generation request, and the knowledge block.

[0203] In one of the embodiments, the text generation module is further configured to:

[0204] Calculate the routing complexity and retrieval complexity of any two nodes in the dynamic label data before and after merging;

[0205] Based on the routing complexity and the retrieval complexity, determine the weighted loss of the dynamic label data;

[0206] If the weighted loss of the two nodes after merging is less than the weighted loss before merging, merge the two nodes.

[0207] In one embodiment, the text generation module is further configured to:

[0208] Obtain the node in the pruning label data that matches the text generation request;

[0209] Obtain the knowledge block under the node that matches the text generation request;

[0210] If the knowledge block under the node that matches the text generation request does not meet the preset condition, select the parent node of the node and obtain the knowledge block under the parent node that matches the text generation request.

[0211] Each module in the knowledge base construction apparatus described above can be realized wholly or partially by software, hardware, and combinations thereof. Each module described above can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0212] In one embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram thereof can be as shown in Figure 7 The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved through WIFI, mobile cellular network, NFC (near field communication), or other technologies. The computer program is executed by the processor to implement a knowledge base construction method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or can be a key, trackball, or touchpad arranged on the shell of the computer device. In addition, the input device can be an external keyboard, touchpad, or mouse, etc.

[0213] Those skilled in the art can understand that Figure 7 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0214] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the knowledge base construction method of any of the above embodiments:

[0215] performing directory structure extraction on the target document to obtain a directory structure tree;

[0216] splitting the target document based on the directory structure tree to obtain a plurality of knowledge blocks;

[0217] when the target document does not exist label data, generating the label data corresponding to the target document based on a preset large language model and the summary text of the target document;

[0218] constructing a target knowledge base based on the label data of at least one of the target documents and the plurality of knowledge blocks; the target knowledge base comprises a document tree composed of the label data of the target document.

[0219] In one embodiment, a computer readable storage medium is provided, having a computer program stored thereon, the computer program being executed by a processor to implement the knowledge base construction method of any of the above embodiments:

[0220] performing directory structure extraction on the target document to obtain a directory structure tree;

[0221] splitting the target document based on the directory structure tree to obtain a plurality of knowledge blocks;

[0222] when the target document does not exist label data, generating the label data corresponding to the target document based on a preset large language model and the summary text of the target document;

[0223] constructing a target knowledge base based on the label data of at least one of the target documents and the plurality of knowledge blocks; the target knowledge base comprises a document tree composed of the label data of the target document.

[0224] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0225] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0226] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0227] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method of constructing a knowledge base, characterized by, The knowledge base construction method comprises: performing directory structure extraction on the target document to obtain a directory structure tree; splitting the target document based on the directory structure tree to obtain a plurality of knowledge blocks; when the target document does not have label data, generating label data corresponding to the target document based on a preset large language model and an abstract text of the target document; constructing a target knowledge base based on the label data of at least one target document and the plurality of knowledge blocks; the target knowledge base comprises a document tree composed of the label data of the target document; after constructing the target knowledge base based on the label data of at least one target document and the plurality of knowledge blocks, the method further comprises: in response to a text generation request, determining a plurality of target knowledge bases corresponding to the text generation request; merging the label data of the plurality of target knowledge bases to obtain a dynamic knowledge base and dynamic label data corresponding to the dynamic knowledge base; performing post-tree pruning on the dynamic label data to obtain pruned label data; obtaining a knowledge block in the pruned label data that matches the text generation request; based on a preset large language model, the text generation request and the knowledge block, obtaining a text generation result corresponding to the text generation; the post-tree pruning of the dynamic label data to obtain pruned label data comprises: calculating the routing complexity and retrieval complexity of any two nodes in the dynamic label data before and after merging; based on the routing complexity and the retrieval complexity, determining the weighted loss of the dynamic label data; if the weighted loss of the two nodes after merging is less than the weighted loss before merging, the two nodes are merged; the routing complexity is an index reflecting the level depth or jump number required from the root node to the target node in the document tree; the retrieval complexity is an index reflecting the number of operations required to find the associated knowledge block through a certain node.

2. The knowledge base construction method according to claim 1, characterized by, The directory structure extraction on the target document to obtain a directory structure tree comprises: extracting a first-level title and a second-level title of the target document based on outline level information, style level information or text matching rules of the target document; if the extraction fails, using the file name of the target document as the first-level title; based on the first-level title, or the first-level title and the second-level title, obtaining the directory structure tree.

3. The knowledge base construction method according to claim 1, characterized by, The directory structure extraction on the target document to obtain a directory structure tree further comprises: converting the target document into a lightweight markup language text; extracting initial title information and an abstract text of the target document based on the lightweight markup language text; generating a directory structure tree corresponding to the target document based on a preset large language model, the initial title information and the abstract text.

4. The knowledge base construction method according to claim 1, characterized by, The splitting of the target document based on the directory structure tree to obtain a plurality of knowledge blocks comprises: based on a preset sliding window and a preset step size, sequentially cutting the text data corresponding to each title in the directory structure tree to obtain a plurality of knowledge blocks.

5. The knowledge base construction method according to claim 1, characterized by, The obtaining of the knowledge block in the pruned label data that matches the text generation request comprises: obtaining a knowledge block in the pruned label data that matches the text generation request; obtaining a knowledge block in the pruned label data that matches the text generation request; if the knowledge block in the pruned label data that matches the text generation request does not meet a preset condition, selecting a parent node of the node and obtaining a knowledge block in the parent node that matches the text generation request.

6. A knowledge base construction apparatus characterized by comprising: The device comprises: a directory extraction module configured to extract a directory structure of a target document to obtain a directory structure tree; a document splitting module configured to split the target document based on the directory structure tree to obtain a plurality of knowledge blocks; a label generation module configured to generate label data corresponding to the target document based on a preset large language model and an abstract text of the target document when the target document does not have label data; a knowledge base construction module configured to construct a target knowledge base based on label data of at least one target document and a plurality of knowledge blocks; the target knowledge base comprises a document tree composed of the label data of the target document; after constructing the target knowledge base based on the label data of at least one target document and a plurality of knowledge blocks, the knowledge base construction module is further configured to: in response to a text generation request, determine a plurality of target knowledge bases corresponding to the text generation request; merge label data of the plurality of target knowledge bases to obtain a dynamic knowledge base and dynamic label data corresponding to the dynamic knowledge base; perform post-tree pruning on the dynamic label data to obtain pruned label data; obtain a knowledge block in the pruned label data that matches the text generation request; and obtain a text generation result corresponding to the text generation based on a preset large language model, the text generation request, and the knowledge block; the post-tree pruning of the dynamic label data to obtain pruned label data comprises: calculating route complexity and retrieval complexity of any two nodes in the dynamic label data before and after merging; determining a weighted loss of the dynamic label data based on the route complexity and the retrieval complexity; if the weighted loss of the two nodes after merging is less than the weighted loss before merging, merging the two nodes; the route complexity is an index reflecting the level depth or the number of jumps required from a root node to a target node in a document tree; the retrieval complexity is an index reflecting the number of operations required to find associated knowledge blocks through a certain node. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The processor executes the computer program to implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 5.

Images