Question and answer method and system based on paragraph splitting and knowledge engine platform

Abstract

The invention relates to the technical field of data processing, and provides a question and answer method and system based on paragraph splitting and a knowledge engine platform, and the question and answer method comprises the steps: carrying out directory and title recognition on an uploaded document, and generating a paragraph frame structure with a hierarchical relationship of the document; splitting the text content of the document according to the paragraph frame structure to obtain a document paragraph structure dictionary containing the corresponding text content; according to the document paragraph structure dictionary, text blocks are generated, and a knowledge base is constructed; performing text block vectorization on the text block by adopting a text vectorization model to generate a sparse vector and a dense vector, and storing the sparse vector and the dense vector in a knowledge base; performing keyword extraction and text vectorization processing on the obtained question to obtain a keyword list, a sparse vector and a dense vector of the question; retrieving the knowledge base, and selecting a plurality of text blocks most similar to the problem; and based on the to-be-queried question, a preset cue word and a plurality of text blocks most similar to the question, obtaining a question answer by adopting the large model.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a question-answering method, system and knowledge engine platform based on paragraph segmentation. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In an organization, a large amount of unstructured document data is usually stored, such as files in Word and PDF formats. However, this data is often not fully utilized, and the potential value of the data is not fully utilized. The main reasons are:

[0004] 1. Low efficiency in knowledge retrieval. When searching for knowledge in existing documents, the system usually only returns a complete document, and employees need to manually search for the corresponding knowledge points in it. This method is not only complicated but also time-consuming, which seriously affects work efficiency.

[0005] 2. Knowledge is relatively scattered. There are information islands within the enterprise. Different departments have different management methods and different ways of storing documents. They cannot communicate with each other. Knowledge management is relatively scattered, which hinders the effective use of data.

[0006] 3. Poor availability of knowledge. At present, the storage form of documents in enterprises may be just a simple document management platform or a document library. In the search and use stage, data exists directly in the form of documents. There are big problems in searching the content in the documents or analyzing the data in them, which greatly reduces the utilization rate of data and cannot give full play to the value of data. Summary of the invention

[0007] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a question-answering method, system and knowledge engine platform based on paragraph splitting, which improves the accuracy of knowledge base question-answering.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A first aspect of the present invention provides a question-answering method based on paragraph segmentation.

[0010] A question-answering method based on paragraph segmentation, comprising:

[0011] Identify the contents and titles of the uploaded documents and generate a paragraph framework structure with hierarchical relationships;

[0012] Split the document body content according to the paragraph frame structure to obtain a document paragraph structure dictionary containing corresponding text content;

[0013] Generate text blocks and build a knowledge base based on the document paragraph structure dictionary;

[0014] The text block is vectorized by using the text vectorization model to generate sparse vectors and dense vectors, which are stored in the knowledge base.

[0015] Perform keyword extraction and text vectorization on the obtained questions to obtain the keyword list, sparse vector and dense vector of the questions; search the knowledge base based on the keyword list, sparse vector and dense vector of the questions respectively, and select several text blocks that are most similar to the questions;

[0016] Based on the question to be queried, the preset prompt words and several text blocks that are most similar to the question, a large model is used to obtain the answer to the question.

[0017] Furthermore, the uploaded document is subjected to catalog and title recognition to generate a paragraph framework structure with a hierarchical relationship; the method includes:

[0018] Extracting information from the uploaded document to obtain first text information;

[0019] Based on the first text information, a directory recognition model is used to obtain directory structure information or output information without directory structure;

[0020] Based on the directory structure information, the document body information is split to obtain the second text information corresponding to the last-level title of each chapter in the directory;

[0021] The first text information or the second text information is split according to line break characters to obtain a plurality of paragraph texts, and the paragraph texts are merged from top to bottom so that the length of the merged text does not exceed a set threshold, thereby obtaining a plurality of third text information;

[0022] Based on the third text information, a title recognition model is used to obtain a title; and the titles are spliced ​​according to the order of the third text information to obtain title structure information of the third text information;

[0023] For documents without directory structure information, a large model is used based on the title structure information and preset prompt words to obtain a paragraph framework structure with a title hierarchy structure;

[0024] For documents with directory structure information, a large model is used based on the directory structure information, the title structure information thereunder and preset prompt words to obtain a paragraph framework structure with a title hierarchy structure.

[0025] Further, based on the directory structure information, the document body information is split to obtain the second text information corresponding to the last level title of each chapter in the directory; the method includes:

[0026] Removing the directory structure information from the first text information to obtain the body information of the document;

[0027] The text information is split according to the line break characters, and each line of text information after the split is vectorized and the text information is embedded;

[0028] Perform text vectorization processing on each title information in the directory structure information to obtain the title information embedding;

[0029] Based on the title information embedding, the most similar text information embedding is searched in the knowledge base as the corresponding title information of the title in the document, and the directory information is updated to obtain new directory information;

[0030] The main text information is matched and split according to the new directory information to obtain the second text information corresponding to the last-level title of each chapter in the directory.

[0031] Furthermore, the body content of the document is split according to the paragraph framework structure to obtain a document paragraph structure dictionary containing corresponding text content; the method includes: converting the paragraph framework structure into an ordered list without a hierarchical structure; splitting the body part information of the document according to the ordered list to obtain text content corresponding to the title; and constructing a document paragraph structure dictionary.

[0032] Furthermore, a text block is generated according to the document paragraph structure dictionary; the method includes: traversing the document paragraph structure dictionary, splicing the titles in the paragraph structure level by level until the last node and the text content corresponding to the last node, and at the same time, judging the theme similarity between the last node and its sibling nodes, if its theme is similar to the theme of the sibling node, then further splicing the sibling node and its corresponding text content with it.

[0033] Further, keyword extraction and text vectorization processing are performed on the obtained questions to obtain a keyword list, a sparse vector and a dense vector of the questions; the knowledge base is searched based on the keyword list, the sparse vector and the dense vector of the questions respectively, and several text blocks most similar to the questions are selected; the method includes:

[0034] Extract keywords from the acquired questions to obtain a keyword list of the questions;

[0035] Perform text vectorization on the obtained questions to obtain sparse vectors and dense vectors of the questions;

[0036] Searching the knowledge base based on the keyword list of the question to obtain a first text block list containing the keywords;

[0037] Search the knowledge base based on the sparse vector of the question to obtain a list of second text blocks that are most similar to the question;

[0038] Search the knowledge base based on the dense vector of the question to obtain a list of third text blocks that are most similar to the question;

[0039] The text blocks in the first text block list, the second text block list and the third text block list are sorted, and several text blocks that are most similar to the question are selected.

[0040] A second aspect of the present invention provides a question-answering system based on paragraph segmentation.

[0041] A question-answering system based on paragraph splitting, comprising:

[0042] A paragraph framework structure generation module is configured to: identify the contents and titles of the uploaded document and generate a paragraph framework structure with a hierarchical relationship of the document;

[0043] A splitting module is configured to: split the document body content according to the paragraph frame structure to obtain a document paragraph structure dictionary containing corresponding text content;

[0044] A text block generation module is configured to: generate text blocks and construct a knowledge base according to a document paragraph structure dictionary;

[0045] A text vectorization processing module is configured to: use a text vectorization model to perform text block vectorization on a text block, generate a sparse vector and a dense vector, and store them in a knowledge base;

[0046] The question processing module is configured to: perform keyword extraction and text vectorization processing on the acquired question to obtain a keyword list, a sparse vector and a dense vector of the question; search the knowledge base based on the keyword list, the sparse vector and the dense vector of the question respectively, and select several text blocks that are most similar to the question;

[0047] The answer generation module is configured to obtain the answer to the question using a large model based on the question to be queried, preset prompt words and several text blocks that are most similar to the question.

[0048] The third aspect of the present invention provides a knowledge engine platform.

[0049] A knowledge engine platform, comprising:

[0050] The data layer is located at the bottom layer of the knowledge engine platform and is used to provide data support for the digital intelligence operator capability layer;

[0051] The digital intelligence operator capability layer is used to select data processing methods based on business needs and upload processing results to the platform layer;

[0052] The platform function layer includes the knowledge production module, the knowledge base construction module and the knowledge service module. The knowledge production module is used to split and analyze documents and extract relevant knowledge from documents; the knowledge base construction module is used to store data; the knowledge service module is used to support intelligent retrieval, knowledge base question and answer, intelligent question counting and graph question and answer services;

[0053] The application layer, based on user input, calls the knowledge production module, knowledge base construction module and / or knowledge service module of the platform layer to perform intelligent retrieval, intelligent reporting, text parsing, intelligent question counting, knowledge base question and answer, and graph question and answer; among which, the knowledge base question and answer process adopts the question and answer method based on paragraph splitting described in the first aspect.

[0054] A fourth aspect of the present invention provides a computer-readable storage medium.

[0055] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the question-answering method based on paragraph splitting as described in the first aspect above.

[0056] A fifth aspect of the present invention provides a computer device.

[0057] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the question-answering method based on paragraph splitting as described in the first aspect above are implemented.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] In practical applications, when retrieving the required knowledge, it is usually necessary to locate the document first, and then retrieve the required knowledge from the document, which will lead to low efficiency in knowledge acquisition. In order to solve this problem, the present invention proposes a question-and-answer method and system based on paragraph splitting. The knowledge base question-and-answer service in the platform can enable users to quickly obtain the required knowledge in a natural language interactive form, solving the problem of low efficiency in users obtaining the required knowledge; on the other hand, when the knowledge base question-and-answer function is implemented in the platform, a paragraph splitting block method is adopted, which solves the problem of the existing knowledge base question-and-answer text block method that divides the text based on character length into blocks, resulting in the separation of the same paragraph text information and low knowledge acquisition accuracy; in addition, in the process of implementing the knowledge base question-and-answer, a knowledge recall method of hybrid retrieval is adopted to solve the problem of incomplete acquisition of question-related background knowledge when a single retrieval method is used, further improving the accuracy of the knowledge base question-and-answer.

[0060] The platform provides componentized capabilities in the three stages of knowledge production, knowledge base construction, and knowledge services, providing business personnel with efficient knowledge extraction and processing of unstructured data, greatly improving the efficiency of data processing; at the same time, business personnel can quickly build a knowledge base for business scenarios, helping customers to quickly build an internal knowledge management system, ensuring centralized management and standardized updating of knowledge, and effectively solving the problems of knowledge dispersion and poor availability; in addition, the platform provides general search, question-answering, reasoning and generation capabilities, allowing users to quickly generate knowledge service APIs combined with specific business scenarios through configuration, making it easier for developers to quickly build knowledge applications for specific businesses. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0062] Figure 1 is a flow chart of a question-answering method based on paragraph splitting shown in the present invention;

[0063] Figure 2 It is a structural diagram of the knowledge engine platform shown in the present invention. DETAILED DESCRIPTION

[0064] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0065] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0066] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0067] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and systems according to various embodiments of the present disclosure. It should be noted that each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code may include one or more executable instructions for implementing the logical functions specified in each embodiment. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram can be implemented using a dedicated hardware-based system that performs a specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0068] Embodiment 1

[0069] like Figure 1 As shown, this embodiment provides a question-and-answer method based on paragraph splitting. This embodiment uses the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In this embodiment, the method includes the following steps:

[0070] Step 1: Select the document to be imported into the knowledge base, such as documents in word, pdf, txt and other formats.

[0071] Step 2: Identify the table of contents and titles of the document and generate a paragraph framework structure with a hierarchical relationship. The specific method includes:

[0072] Step 2.1: Preprocess the uploaded documents and extract text information from documents in PDF, Word and other formats to obtain text information.

[0073] Step 2.2: Use the directory recognition model to identify the directory in the document information. If the document contains directory information, return the directory structure information and execute step 2.3. If not, it means that there is no directory information in the document and execute step 2.4.

[0074] Step 2.3: According to the directory structure information identified in step 2.2, the document body information is split to obtain the text information corresponding to the last-level title of each chapter in the directory. The specific method includes:

[0075] Step 2.3.1: From the text information obtained in step 2.1, remove the directory information identified in step 2.2 to obtain the main text information in the document information.

[0076] Step 2.3.2: The text information obtained in step 2.3.1 is split according to line breaks, and each line of text information after the split is vectorized using a text vectorization model and stored in a vector database.

[0077] Step 2.3.3: Each title information in the directory structure information identified in step 2.2 is embedded using the text vectorization model used in step 2.3.2.

[0078] Step 2.3.4: Retrieve the most similar text information from the vector library for each embedded title information as the corresponding title information in the text of the directory title and update the directory title information to obtain new directory information. This method can effectively solve the problem of inconsistency between the title in the directory and the title in the text, and the subsequent inability to match the text.

[0079] Step 2.3.5: record the updated directory information as list_columns, match and split the document body information according to the list_columns directory information, and obtain the text information corresponding to the last level title of each chapter in the directory. Wherein, the list_columns data format is: list_columns = [("Chapter 1 XXX", 1), ("Chapter 2 XXX", 1), ("2.1XXX", 2), ("2.2XXX", 2)], ("Chapter 1 XXX", 1) The first element is the name, and the second element is the level.

[0080] The specific splitting method is: get the maximum number of levels of the directory and record it as level, traverse the list_columns list, record the element with the level value of level as the current element and record it as item_current, and get the next element of the current element and record it as item_next, then split the body text according to the line break to get the body list, and locate the item_current and item_next elements from the body list. In order to ensure that the located element is a title element, the located information is further verified according to the title rules, and then the text content between item_current and item_next is intercepted, which is the text content corresponding to the title item item_current. Then, for the text information corresponding to the last-level title of each directory obtained, continue to perform the following steps.

[0081] Step 2.4: Split the text information according to the line break, and record the text paragraph list after the split as paras. Then, merge the split paragraph texts from top to bottom until the text length threshold (30,000 characters) is reached. Use this method to split the text information into multiple text information to form a text information list. The example is as follows:

[0082] paras=[p 1 ,p 2 ,p 3 ,p 4 ,p 5 ,p 6 ......,p n-2 ,p n-1 ,p n ], if len(p 1 )+len(p 2 )<30000 and

[0083] len(p 1 )+len(p 2 )+len(p 3 )>30000, then p 1 With p 2 Merge into a text information block1, and then from p 3 Start, continue to judge, if len(p 3 )+len(p 4 )<30000 and len(p 3 )+len(p 4 )+len(p 5 )>30000, then p 3 With p 4 Merge into a text message block2, if len(p 3)+len(p 4 )<30000 and len(p 3 )+len(p 4 )+len(p 5 )<30000, then continue to judge len(p 3 )+len(p 4 )+len(p 5 )+len(p 6 ), if the length is greater than the threshold, p 3 、p 4 、p 5 Merge into one text message, and so on, eventually forming a text message list.

[0084] Step 2.5: Use the pre-trained title recognition model to recognize the title in each text message after the split, and then splice the title obtained from each text message in the order of the text information to obtain the title structure of the text information.

[0085] Step 2.6: Generate the paragraph framework structure of the document based on the title structure information obtained in step 2.5.

[0086] The specific method is as follows:

[0087] For documents without a table of contents, the title structure obtained in step 2.5 is used as the input information of the big model through the understanding ability of the big model, and the title structure information and the set prompt word information are used to obtain a paragraph framework structure with a title hierarchy structure.

[0088] For documents with a directory, based on the document directory information obtained in step 2.3.4, the text information corresponding to the last-level title of each chapter in the directory obtained in step 2.3.5, and the sub-title structure information corresponding to the directory title obtained in steps 2.4 and 2.5, the large model's understanding ability is used to take the document directory information and its sub-title structure information and the set prompt word information as input information for the large model to obtain the paragraph framework structure of the document with a title hierarchy structure.

[0089] Step 3: Split the document body content according to the document paragraph framework structure generated in step 2 to obtain a document paragraph structure dictionary containing corresponding text content. The specific method includes:

[0090] Step 3.1: Convert the document paragraph frame structure into an ordered list without a hierarchical structure. For example:

[0091] {

[0092] "Chapter 1 XXX": [],

[0093] "Chapter 2 XXX": [

[0094] {"2.1XXX":[]},

[0095] {"2.2XXX": []}

[0096] ],

[0097] "Chapter 3 XXX": [

[0098] {"3.1XXX":[]},

[0099] {"3.2XXX": []}

[0100] ],

[0101] }

[0102] After conversion:

[0103] ["Chapter 1 XXX", "Chapter 2 XXX", "2.1XXX", "2.2XXX", "Chapter 3 XXX", "3.1XXX", "3.2XXX"], recorded as list_titles.

[0104] Step 3.2: Split the document body content according to the ordered list to obtain the text content corresponding to the title item.

[0105] The specific splitting method is: traverse the list_titles list, get the current element and record it as item_current, and get the next element of the current element and record it as item_next. Then, split the body text according to the line break to get the body list, and locate the item_current and item_next elements from the body list. In order to ensure that the located element is a title element, the located information is further verified according to the title rules, and then the text content between item_current and item_next is intercepted as the body content corresponding to the title item item_current, and finally the document paragraph structure dictionary containing the corresponding text content is obtained. The example is as follows:

[0106] The text content is:

[0107] Chapter 2 XXXX

[0108] XXXX Transition

[0109] 2.1XXXX

[0110] XXXX Content 2.1

[0111] 2.2XXXX

[0112] The text content dictionary corresponding to the XXXX content 2.2 title is:

[0113]

[0114] }

[0115] }

[0116] Step 4: Generate text blocks based on the document paragraph structure dictionary split in step 3.

[0117] The specific steps are: traverse the document paragraph structure dictionary containing the corresponding text content obtained in step 3, and splice the titles in the paragraph structure level by level until the last node and the text content corresponding to the last node. At the same time, judge the theme similarity between the last node and its sibling nodes (titles of the same level under the same title). If its theme is similar to that of the sibling node, the sibling node and its corresponding text content will be further spliced ​​with it.

[0118] It is worth noting that when the titles in the paragraph structure are spliced ​​here, the transition paragraph content corresponding to the non-final title node is also spliced. This method can enrich the semantic information of the final spliced ​​text block, make the retrieval of question-related text blocks more accurate during knowledge base question and answer, and improve the accuracy of question and answer. The following is an example of the spliced ​​text block:

[0119] Case 1: The sibling node has a different topic from the current node. The final text block after concatenation is as follows:

[0120] Final text block 1:

[0121] Chapter 2XXX

[0122] XXXX Transition

[0123] 2.1XXX

[0124] XXXX Content 2.1

[0125] Final text block 2:

[0126] Chapter 2XXX

[0127] XXXX Transition

[0128] 2.1XXX

[0129] XXXX Content 2.2

[0130] Case 2: The sibling node has a similar topic to the current node. The final text block after concatenation is as follows:

[0131] Final text block 1:

[0132] Chapter 2XXX

[0133] XXXX Transition

[0134] 2.1XXX

[0135] XXXX Content 2.1

[0136] 2.1XXX

[0137] XXXX Content 2.2

[0138] At this point, the text block is generated.

[0139] Step 5: The text blocks generated in the above steps are vectorized using a text vectorization model and stored in a vector database to form a knowledge base.

[0140] The specific method is: use the text vectorization model to generate sparse vectors and dense vectors for the text block, and then store the text block and the two types of vector data into the text field and the sparse and dense vector fields of the vector library to form a knowledge base.

[0141] Step 6: Based on the questions raised by the user, a hybrid retrieval method is used to retrieve relevant text block information from the knowledge base. The specific methods include:

[0142] Step 6.1: Extract keywords from the questions raised by the user to obtain a list of keywords in the questions.

[0143] Step 6.2: Use the text vectorization model used in the above step to obtain the sparse vector and dense vector of the question asked by the user.

[0144] Step 6.3: According to the keywords in the question, retrieve the text blocks containing the keywords from the knowledge base and record them as blocks1.

[0145] Step 6.4: Use the sparse vector of the question to obtain the top N text blocks most similar to the question from the knowledge base according to the sparse vector field, recorded as blocks2.

[0146] Step 6.5: Use the dense vector of the question to obtain the top N text blocks most similar to the question from the knowledge base according to the dense vector field, recorded as blocks3.

[0147] Step 6.6: Reorder the text block information obtained in step 6.3, step 6.4 and step 6.5 using the reordering algorithm of the vector library, and then obtain the top N text block information that is most similar to the question.

[0148] Step 7: Generate the required answer based on the question raised by the user, the set prompt words and the recalled text block information.

[0149] The specific method is: concatenate the question raised by the user with the text block information related to the question obtained in step 6 and the prompt word information as the input of the large model, and use the understanding ability of the large model to return the answer information corresponding to the question.

[0150] In the present invention, the directory recognition model, text vectorization model, title recognition model, etc. can adopt existing machine learning algorithms, which can be based on neural networks, support vector machine regression and other algorithms. The embodiments of the present application are not limited to this.

[0151] The implementation of the knowledge base question and answer service in the present invention is based on the paragraph splitting and block division method and the hybrid retrieval text block recall method. On the one hand, it solves the problem of breaking up the same paragraph text information due to the text block division based on character length when constructing the existing knowledge base, thereby improving the accuracy of knowledge acquisition; on the other hand, in the process of implementing the knowledge base question and answer, the hybrid retrieval knowledge recall method is adopted to solve the problem of incomplete acquisition of question-related background knowledge when a single retrieval method is adopted, thereby further improving the accuracy of the knowledge base question and answer. In addition, the knowledge base question and answer service allows users to quickly obtain the required knowledge in the form of natural language interaction, thereby improving the efficiency of knowledge acquisition.

[0152] Embodiment 2

[0153] This embodiment provides a question-answering system based on paragraph segmentation.

[0154] A question-answering system based on paragraph splitting, comprising:

[0155] A paragraph framework structure generation module is configured to: identify the contents and titles of the uploaded document and generate a paragraph framework structure with a hierarchical relationship of the document;

[0156] A splitting module is configured to: split the document body content according to the paragraph frame structure to obtain a document paragraph structure dictionary containing corresponding text content;

[0157] A text block generation module is configured to: generate text blocks and construct a knowledge base according to a document paragraph structure dictionary;

[0158] A text vectorization processing module is configured to: use a text vectorization model to perform text block vectorization on a text block, generate a sparse vector and a dense vector, and store them in a knowledge base;

[0159] The question processing module is configured to: perform keyword extraction and text vectorization processing on the acquired question to obtain a keyword list, a sparse vector and a dense vector of the question; search the knowledge base based on the keyword list, the sparse vector and the dense vector of the question respectively, and select several text blocks that are most similar to the question;

[0160] The answer generation module is configured to obtain the answer to the question using a large model based on the question to be queried, preset prompt words and several text blocks that are most similar to the question.

[0161] In some embodiments, the paragraph framework structure generation module is further configured to: extract information from the uploaded document to obtain first text information; based on the first text information, use a directory recognition model to obtain directory structure information or output information without directory structure; based on the directory structure information, split the document body information to obtain second text information corresponding to the last-level title of each chapter in the directory; split the first text information or the second text information according to line break characters to obtain a number of paragraph texts, merge the paragraph texts from top to bottom, and the length of the merged text does not exceed a set threshold to obtain a number of third text information; based on the third text information, use a title recognition model to obtain a title; and splice the titles in the order of the third text information to obtain title structure information of the third text information; for documents without directory structure information, use a large model based on the title structure information and preset prompt words to obtain a paragraph framework structure with a title hierarchy structure; for documents with directory structure information, use a large model based on the directory structure information, the title structure information thereunder, and the preset prompt words to obtain a paragraph framework structure with a title hierarchy structure.

[0162] In some embodiments, the paragraph frame structure generation module is further configured to: remove the directory structure information from the first text information to obtain the main text information of the document; split the main text information according to the line break characters, and perform text vectorization processing on each line of text information after the split, and embed the text information; perform text vectorization processing on each title information in the directory structure information to obtain title information embedding; based on the title information embedding, query the most similar text information embedding in the knowledge base as the title information corresponding to the title in the document, and update the directory information to obtain new directory information; match and split the main text information according to the new directory information to obtain the second text information corresponding to the last-level title of each chapter in the directory.

[0163] The splitting module is also configured to: convert the paragraph frame structure into an ordered list without a hierarchical structure; split the body information of the document according to the ordered list to obtain the text content corresponding to the title; and construct a document paragraph structure dictionary.

[0164] The text block generation module is also configured to: traverse the document paragraph structure dictionary, splice the titles in the paragraph structure level by level until the last node and the text content corresponding to the last node, and at the same time, judge the theme similarity between the last node and its sibling nodes. If its theme is similar to that of the sibling node, the sibling node and its corresponding text content will be further spliced ​​with it.

[0165] The question processing module is also configured to: extract keywords from the acquired questions to obtain a keyword list of the questions; perform text vectorization on the acquired questions to obtain a sparse vector and a dense vector of the questions; search the knowledge base based on the keyword list of the questions to obtain a first text block list containing keywords; search the knowledge base based on the sparse vectors of the questions to obtain a second text block list that is most similar to the questions; search the knowledge base based on the dense vectors of the questions to obtain a third text block list that is most similar to the questions; sort the text blocks in the first text block list, the second text block list, and the third text block list, and select several text blocks that are most similar to the questions.

[0166] Embodiment 3

[0167] like Figure 2 As shown, this embodiment provides a knowledge engine platform, which is based on big models and knowledge graphs and includes three links: knowledge production, knowledge base construction, and knowledge services.

[0168] The overall architecture of the knowledge engine platform includes the data layer, digital operator capability layer, platform function layer and application layer from bottom to top.

[0169] Data layer: The data layer is located at the bottom layer of the platform and provides data support for upper layer applications. The data source can be structured data, semi-structured data, unstructured data, or other data sources.

[0170] Digital intelligence operator capability layer: This layer brings together data intelligence capabilities in a modular form. It includes data intelligence capabilities such as knowledge extraction, intelligent labeling, text2sql, intent recognition, question rewriting, text vectorization, text2cypher, text recall & sorting, and subgraph retrieval. Different capability combinations are selected according to different business needs to implement corresponding functions and provide different services for upper-layer applications.

[0171] Platform functional layer: The platform consists of three major modules, namely knowledge production, knowledge base, and knowledge services.

[0172] The knowledge production module mainly uses a series of intelligent data processing components based on large models to disassemble and analyze unstructured document data in different formats such as word, pdf, txt, etc., and extract useful knowledge from the documents.

[0173] The knowledge base construction module mainly stores, manages and updates knowledge in different knowledge representation forms. The platform provides three different knowledge storage forms, including structured database, vector database and knowledge graph. Through the knowledge base construction link, business personnel can quickly build a knowledge base management system within the organization, make knowledge management more standardized, and solve problems such as knowledge dispersion and poor availability within the organization.

[0174] The knowledge service module is mainly based on general intelligent capabilities such as search, reasoning, question-answering, and generation, enabling business personnel to quickly generate corresponding knowledge service APIs based on the needs of specific business scenarios, and then create knowledge applications for specific scenarios. Currently, it supports intelligent retrieval, knowledge base question-answering, intelligent question counting, and graph question-answering services.

[0175] The specific core functional modules of the platform functional layer are as follows:

[0176] Data source management: realize unified management of database connection information, support access to MySQL, TiDB relational database data sources and Minio data sources.

[0177] Data management: realize unified uploading and management of unstructured document data (word, pdf, txt).

[0178] Knowledge extraction: Analyze unstructured document data based on intelligent data processing components, extract useful knowledge from the documents and store it in the database.

[0179] Knowledge graph construction: Define the schema of the knowledge graph by dragging and dropping, and map entities, attributes, and relationships to generate a graph.

[0180] Graph visualization: Visualize the graph to facilitate users to quickly search and view the data information in the graph.

[0181] Knowledge base management: realizes functions such as knowledge base creation and knowledge storage, and supports both structured knowledge base and unstructured knowledge base formats.

[0182] Knowledge service: Through configuration, business-specific knowledge service APIs are quickly generated to support services such as intelligent retrieval, knowledge base question and answer, intelligent question counting, and graph question and answer.

[0183] Model management: Manage a series of data intelligence operator models based on large models and natural language processing technologies, and support the addition and deletion of models.

[0184] Application layer: This layer provides users with the required services based on the knowledge base and digital intelligence, including intelligent retrieval, intelligent reporting, text analysis, intelligent question and answer, knowledge base question and answer, graph question and answer, etc. At the same time, it supports the development of customized services.

[0185] In the process of platform construction, in order to enable users to obtain the required knowledge from the knowledge base more efficiently and accurately, the present invention adopts the block division method of paragraph splitting and the background knowledge retrieval method of hybrid retrieval to realize knowledge base question answering. The specific implementation steps are as follows:

[0186] Step 1: Select the document to be imported into the knowledge base, such as documents in word, pdf, txt and other formats.

[0187] Step 2: Identify the table of contents and titles of the document and generate a paragraph framework structure with a hierarchical relationship. The specific method includes:

[0188] Step 2.1: Preprocess the uploaded documents and extract text information from documents in PDF, Word and other formats to obtain text information.

[0189] Step 2.2: Use the directory recognition model to identify the directory in the document information. If the document contains directory information, return the directory structure information and execute step 2.3. If not, it means that there is no directory information in the document and execute step 2.4.

[0190] Step 2.3: According to the directory structure information identified in step 2.2, the document body information is split to obtain the text information corresponding to the last-level title of each chapter in the directory. The specific method includes:

[0191] Step 2.3.1: From the text information obtained in step 2.1, remove the directory information identified in step 2.2 to obtain the main text information in the document information.

[0192] Step 2.3.2: The text information obtained in step 2.3.1 is split according to line breaks, and each line of text information after the split is vectorized using a text vectorization model and stored in a vector database.

[0193] Step 2.3.3: Each title information in the directory structure information identified in step 2.2 is embedded using the text vectorization model used in step 2.3.2.

[0194] Step 2.3.4: Retrieve the most similar text information from the vector library for each embedded title information as the corresponding title information in the text of the directory title and update the directory title information to obtain new directory information. This method can effectively solve the problem of inconsistency between the title in the directory and the title in the text, and the subsequent inability to match the text.

[0195] Step 2.3.5: record the updated directory information as list_columns, match and split the document body information according to the list_columns directory information, and obtain the text information corresponding to the last level title of each chapter in the directory. Wherein, the list_columns data format is: list_columns = [("Chapter 1 XXX", 1), ("Chapter 2 XXX", 1), ("2.1XXX", 2), ("2.2XXX", 2)], ("Chapter 1 XXX", 1) The first element is the name, and the second element is the level.

[0196] The specific splitting method is: get the maximum number of levels of the directory and record it as level, traverse the list_columns list, record the element with the level value of level as the current element and record it as item_current, and get the next element of the current element and record it as item_next, then split the body text according to the line break to get the body list, and locate the item_current and item_next elements from the body list. In order to ensure that the located element is a title element, the located information is further verified according to the title rules, and then the text content between item_current and item_next is intercepted, which is the text content corresponding to the title item item_current. Then, for the text information corresponding to the last-level title of each directory obtained, continue to perform the following steps.

[0197] Step 2.4: Split the text information according to the line break, and record the text paragraph list after the split as paras. Then, merge the split paragraph texts from top to bottom until the text length threshold (30,000 characters) is reached. Use this method to split the text information into multiple text information to form a text information list. The example is as follows:

[0198] paras=[p 1 ,p 2 ,p 3 ,p 4 ,p 5 ,p 6 ......,p n-2 ,p n-1 ,p n ], if len(p 1 )+len(p 2 )<30000 and

[0199] len(p 1 )+len(p 2 )+len(p 3 )>30000, then p 1 With p2 Merge into a text information block1, and then from p 3 Start, continue to judge, if len(p 3 )+len(p 4 )<30000 and len(p 3 )+len(p 4 )+len(p 5 )>30000, then p 3 With p 4 Merge into a text message block2, if len(p 3 )+len(p 4 )<30000 and len(p 3 )+len(p 4 )+len(p 5 )<30000, then continue to judge len(p 3 )+len(p 4 )+len(p 5 )+len(p 6 ), if the length is greater than the threshold, p 3 、p 4 、p 5 Merge into one text message, and so on, eventually forming a text message list.

[0200] Step 2.5: Use the pre-trained title recognition model to recognize the title in each text message after the split, and then splice the title obtained from each text message in the order of the text information to obtain the title structure of the text information.

[0201] Step 2.6: Generate the paragraph framework structure of the document based on the title structure information obtained in step 2.5.

[0202] The specific method is as follows:

[0203] For documents without a table of contents, the title structure obtained in step 2.5 is used as the input information of the big model through the understanding ability of the big model, and the title structure information and the set prompt word information are used to obtain a paragraph framework structure with a title hierarchy structure.

[0204] For documents with a directory, based on the document directory information obtained in step 2.3.4, the text information corresponding to the last-level title of each chapter in the directory obtained in step 2.3.5, and the sub-title structure information corresponding to the directory title obtained in steps 2.4 and 2.5, the large model's understanding ability is used to take the document directory information and its sub-title structure information and the set prompt word information as input information for the large model to obtain the paragraph framework structure of the document with a title hierarchy structure.

[0205] Step 3: Split the document body content according to the document paragraph framework structure generated in step 2 to obtain a document paragraph structure dictionary containing corresponding text content. The specific method includes:

[0206] Step 3.1: Convert the document paragraph frame structure into an ordered list without a hierarchical structure. For example:

[0207] {

[0208] "Chapter 1 XXX": [],

[0209] "Chapter 2 XXX": [

[0210] {"2.1XXX":[]},

[0211] {"2.2XXX": []}

[0212] ],

[0213] "Chapter 3 XXX": [

[0214] {"3.1XXX":[]},

[0215] {"3.2XXX": []}

[0216] ],

[0217] }

[0218] After conversion:

[0219] ["Chapter 1 XXX", "Chapter 2 XXX", "2.1XXX", "2.2XXX", "Chapter 3 XXX", "3.1XXX", "3.2XXX"], recorded as list_titles.

[0220] Step 3.2: Split the document body content according to the ordered list to obtain the text content corresponding to the title item.

[0221] The specific splitting method is: traverse the list_titles list, get the current element and record it as item_current, and get the next element of the current element and record it as item_next. Then, split the body text according to the line break to get the body list, and locate the item_current and item_next elements from the body list. In order to ensure that the located element is a title element, the located information is further verified according to the title rules, and then the text content between item_current and item_next is intercepted as the body content corresponding to the title item item_current, and finally the document paragraph structure dictionary containing the corresponding text content is obtained. The example is as follows:

[0222] The text content is:

[0223] Chapter 2 XXXX

[0224] XXXX Transition

[0225] 2.1XXXX

[0226] XXXX Content 2.1

[0227] 2.2XXXX

[0228] XXXX Content 2.2

[0229] The text content dictionary corresponding to the title is:

[0230]

[0231]

[0232] Step 4: Generate text blocks based on the document paragraph structure dictionary split in step 3.

[0233] The specific steps are: traverse the document paragraph structure dictionary containing the corresponding text content obtained in step 3, and splice the titles in the paragraph structure level by level until the last node and the text content corresponding to the last node. At the same time, judge the theme similarity between the last node and its sibling nodes. If its theme is similar to that of the sibling node, the sibling node and its corresponding text content are further spliced ​​with it.

[0234] It is worth noting that when the titles in the paragraph structure are spliced ​​here, the transition paragraph content corresponding to the non-final title node is also spliced. This method can enrich the semantic information of the final spliced ​​text block, make the retrieval of question-related text blocks more accurate during knowledge base question and answer, and improve the accuracy of question and answer. The following is an example of the spliced ​​text block:

[0235] Case 1: The sibling node has a different topic from the current node. The final text block after concatenation is as follows:

[0236] Final text block 1:

[0237] Chapter 2XXX

[0238] XXXX Transition

[0239] 2.1XXX

[0240] XXXX Content 2.1

[0241] Final text block 2:

[0242] Chapter 2XXX

[0243] XXXX Transition

[0244] 2.1XXX

[0245] XXXX Content 2.2

[0246] Case 2: The sibling node has a similar topic to the current node. The final text block after concatenation is as follows:

[0247] Final text block 1:

[0248] Chapter 2XXX

[0249] XXXX Transition

[0250] 2.1XXX

[0251] XXXX Content 2.1

[0252] 2.1XXX

[0253] XXXX Content 2.2

[0254] At this point, the text block is generated.

[0255] Step 5: The text blocks generated in the above steps are vectorized using a text vectorization model and stored in a vector database to form a knowledge base.

[0256] The specific method is: use the text vectorization model to generate sparse vectors and dense vectors for the text block, and then store the text block and the two types of vector data into the text field and the sparse and dense vector fields of the vector library to form a knowledge base.

[0257] Step 6: Based on the questions raised by the user, a hybrid retrieval method is used to retrieve relevant text block information from the knowledge base. The specific methods include:

[0258] Step 6.1: Extract keywords from the questions raised by the user to obtain a list of keywords in the questions.

[0259] Step 6.2: Use the text vectorization model used in the above step to obtain the sparse vector and dense vector of the question asked by the user.

[0260] Step 6.3: According to the keywords in the question, retrieve the text blocks containing the keywords from the knowledge base and record them as blocks1.

[0261] Step 6.4: Use the sparse vector of the question to obtain the top N text blocks most similar to the question from the knowledge base according to the sparse vector field, recorded as blocks2.

[0262] Step 6.5: Use the dense vector of the question to obtain the top N text blocks most similar to the question from the knowledge base according to the dense vector field, recorded as blocks3.

[0263] Step 6.6: Reorder the text block information obtained in step 6.3, step 6.4 and step 6.5 using the reordering algorithm of the vector library, and then obtain the top N text block information that is most similar to the question.

[0264] Step 7: Generate the required answer based on the question raised by the user, the set prompt words and the recalled text block information.

[0265] The specific method is: concatenate the question raised by the user with the text block information related to the question obtained in step 6 and the prompt word information as the input of the large model, and use the understanding ability of the large model to return the answer information corresponding to the question.

[0266] Embodiment 4

[0267] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the question-answering method based on paragraph splitting as described in the first embodiment above are implemented.

[0268] Embodiment 5

[0269] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the question-and-answer method based on paragraph splitting as described in the first embodiment above are implemented.

[0270] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A question-answering method based on paragraph splitting, characterized in that: include: Identify the contents and titles of the uploaded documents and generate a paragraph framework structure with hierarchical relationships; Split the document body content according to the paragraph frame structure to obtain a document paragraph structure dictionary containing corresponding text content; Generate text blocks and build a knowledge base based on the document paragraph structure dictionary; The text block is vectorized by using the text vectorization model to generate sparse vectors and dense vectors, which are stored in the knowledge base. Perform keyword extraction and text vectorization on the obtained questions to obtain the keyword list, sparse vector and dense vector of the questions; search the knowledge base based on the keyword list, sparse vector and dense vector of the questions respectively, and select several text blocks that are most similar to the questions; Based on the question to be queried, the preset prompt words and several text blocks that are most similar to the question, a large model is used to obtain the answer to the question.

2. The question-answering method based on paragraph splitting according to claim 1, characterized in that: Identify the directory and title of the uploaded document and generate a paragraph framework structure with a hierarchical relationship for the document; the method includes: Extracting information from the uploaded document to obtain first text information; Based on the first text information, a directory recognition model is used to obtain directory structure information or output information without directory structure; Based on the directory structure information, the document body information is split to obtain the second text information corresponding to the last-level title of each chapter in the directory; The first text information or the second text information is split according to line break characters to obtain a plurality of paragraph texts, and the paragraph texts are merged from top to bottom so that the length of the merged text does not exceed a set threshold, thereby obtaining a plurality of third text information; Based on the third text information, a title recognition model is used to obtain a title; and the titles are spliced ​​according to the order of the third text information to obtain title structure information of the third text information; For documents without directory structure information, a large model is used based on the title structure information and preset prompt words to obtain a paragraph framework structure with a title hierarchy structure; For documents with directory structure information, a large model is used based on the directory structure information, the title structure information thereunder and preset prompt words to obtain a paragraph framework structure with a title hierarchy structure.

3. The question-answering method based on paragraph splitting according to claim 2, characterized in that: Based on the directory structure information, the document body information is split to obtain the second text information corresponding to the last level title of each chapter in the directory; the method includes: Removing the directory structure information from the first text information to obtain the body information of the document; The text information is split according to the line break characters, and each line of text information after the split is vectorized and the text information is embedded; Perform text vectorization processing on each title information in the directory structure information to obtain the title information embedding; Based on the title information embedding, the most similar text information embedding is searched in the knowledge base as the corresponding title information of the title in the document, and the directory information is updated to obtain new directory information; The main text information is matched and split according to the new directory information to obtain the second text information corresponding to the last-level title of each chapter in the directory.

4. The question-answering method based on paragraph splitting according to claim 1, characterized in that: The document body content is split according to the paragraph frame structure to obtain a document paragraph structure dictionary containing corresponding text content; the method includes: converting the paragraph frame structure into an ordered list without a hierarchical structure; splitting the document body part information according to the ordered list to obtain the text content corresponding to the title; and constructing a document paragraph structure dictionary.

5. The question-answering method based on paragraph splitting according to claim 1, characterized in that: Generate a text block according to the document paragraph structure dictionary; the method includes: traversing the document paragraph structure dictionary, splicing the titles in the paragraph structure level by level until the last node and the text content corresponding to the last node, and at the same time, judging the theme similarity between the last node and its brother node, if its theme is similar to the theme of the brother node, then further splicing the brother node and its corresponding text content with it.

6. The question-answering method based on paragraph splitting according to claim 1, characterized in that: Perform keyword extraction and text vectorization on the obtained questions to obtain the keyword list, sparse vector and dense vector of the questions; The knowledge base is searched based on the keyword list, sparse vector and dense vector of the question, and several text blocks that are most similar to the question are selected; the method includes: Extract keywords from the acquired questions to obtain a keyword list of the questions; Perform text vectorization on the obtained questions to obtain sparse vectors and dense vectors of the questions; Searching the knowledge base based on the keyword list of the question to obtain a first text block list containing the keywords; Search the knowledge base based on the sparse vector of the question to obtain a list of second text blocks that are most similar to the question; Search the knowledge base based on the dense vector of the question to obtain a list of third text blocks that are most similar to the question; The text blocks in the first text block list, the second text block list and the third text block list are sorted, and several text blocks that are most similar to the question are selected.

7. A question-answering system based on paragraph splitting, characterized in that: include: A paragraph framework structure generation module is configured to: identify the contents and titles of the uploaded document and generate a paragraph framework structure with a hierarchical relationship of the document; A splitting module is configured to: split the document body content according to the paragraph frame structure to obtain a document paragraph structure dictionary containing corresponding text content; A text block generation module is configured to: generate text blocks and construct a knowledge base according to a document paragraph structure dictionary; A text vectorization processing module is configured to: use a text vectorization model to perform text block vectorization on a text block, generate a sparse vector and a dense vector, and store them in a knowledge base; The question processing module is configured to: perform keyword extraction and text vectorization processing on the acquired question to obtain a keyword list, a sparse vector and a dense vector of the question; search the knowledge base based on the keyword list, the sparse vector and the dense vector of the question respectively, and select several text blocks that are most similar to the question; The answer generation module is configured to obtain the answer to the question using a large model based on the question to be queried, preset prompt words and several text blocks that are most similar to the question.

8. A knowledge engine platform, characterized in that: include: The data layer is located at the bottom layer of the knowledge engine platform and is used to provide data support for the digital intelligence operator capability layer; The digital intelligence operator capability layer is used to select data processing methods based on business needs and upload processing results to the platform layer; The platform function layer includes the knowledge production module, the knowledge base construction module and the knowledge service module. The knowledge production module is used to split and analyze documents and extract relevant knowledge from documents; the knowledge base construction module is used to store data; the knowledge service module is used to support intelligent retrieval, knowledge base question and answer, intelligent question counting and graph question and answer services; The application layer, based on user input, calls the knowledge production module, knowledge base construction module and / or knowledge service module of the platform layer to perform intelligent retrieval, intelligent reporting, text parsing, intelligent question counting, knowledge base question and answer, and graph question and answer; wherein the knowledge base question and answer process adopts the paragraph splitting based question and answer method described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the question-answering method based on paragraph splitting as described in any one of claims 1 to 6 are implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the question-answering method based on paragraph splitting as described in any one of claims 1 to 6 are implemented.

Images