Knowledge base construction method and system for unstructured documents and large model question and answer method and system

Abstract

The invention discloses a knowledge base construction method and system for unstructured documents and a large model question and answer method and system. The method comprises the following steps: acquiring an unstructured document, and dividing document regions; analyzing the document area to obtain unstructured document information; processing the unstructured document information by using a heuristic rule to obtain an initial text; proofreading and modifying the initial text to obtain a final text; vectorizing the final text to obtain a final text vector; and taking the corresponding final text vector and the final text as knowledge base entries, and storing the knowledge base entries as a database. According to the method, the unstructured document can be simply and efficiently processed and the knowledge base can be constructed, so that the accuracy of a large language model question-answering system is improved.

Description

Technical Field

[0001] The present invention relates to the field of knowledge base construction, and more specifically, to a knowledge base construction method and system for unstructured documents and a large model question and answer method and system. Background Art

[0002] Nowadays, technologies related to big language models are developing rapidly, and various companies are trying to apply big language models to business systems to assist business growth. Generally speaking, companies use big language models to build internal knowledge question-and-answer systems to simplify the barriers to internal information sharing and improve the efficiency of overall collaborative office work; or they open up some knowledge to customers and partners to reduce the difficulty for customers and partners to obtain corresponding support.

[0003] In the process of building an intelligent question-answering system related to a large language model, relevant business documents within the enterprise are generally integrated as data sources; most of these are unstructured documents such as PDF, which involve unstructured text content, tables, charts, etc., so it is necessary to perform data cleaning operations on these documents in advance and convert them into a format that can be read and used by the question-answering system. However, traditional methods are difficult to accurately and efficiently extract information from unstructured documents; if the extracted information is used in the knowledge base of a large language model, it also needs to be further processed, and the process is relatively complicated and inefficient.

[0004] The prior art discloses a document processing method, system, device and storage medium. In the method, a local knowledge base stores multiple information documents, and the information documents are preprocessed and split according to a preset document splitting rule to obtain a target document paragraph, which is stored in the local knowledge base and associated with the corresponding information document; the document processing method includes obtaining a knowledge acquisition question input by a user, vectorizing the knowledge acquisition question and extracting feature information to obtain a knowledge acquisition question vector and corresponding vector feature information; based on the knowledge acquisition question vector and the corresponding vector feature information, relevant documents are recalled in the local knowledge base to obtain an initial recall document matching the knowledge acquisition question; based on a preset knowledge acquisition strategy, information is screened and extracted from the initial recall document to obtain an answer to the knowledge acquisition question and a corresponding target knowledge document. The method does not perform targeted extraction processing in combination with the characteristics of a large language model, nor does it correct the extracted information. Summary of the invention

[0005] In view of the defects of low efficiency and complex process of unstructured document extraction for building knowledge base in the prior art, the present invention provides a knowledge base construction method and system for unstructured documents and a large model question and answer method and system, which can simply and efficiently process unstructured documents and build a knowledge base, thereby improving the accuracy of the large language model question and answer system.

[0006] The primary purpose of the present invention is to solve the above technical problems. The technical solutions of the present invention are as follows:

[0007] A method for constructing a knowledge base for unstructured documents, comprising:

[0008] S101: Obtain unstructured documents and divide document areas;

[0009] S102: parsing the document area to obtain unstructured document information;

[0010] S103: Processing the unstructured document information using heuristic rules to obtain an initial text;

[0011] S104: proofreading and modifying the initial text to obtain a final text;

[0012] S105: vectorizing the final text to obtain a final text vector;

[0013] S106: The corresponding final text vector and the final text are taken as knowledge base entries and saved as a database.

[0014] Furthermore, the document information includes: the content of text arranged in lines, the font size of the text in each line, and the coordinates of the location of each line and text.

[0015] Furthermore, the heuristic rules include: one or more of paragraph division, content filtering, and title recognition.

[0016] Furthermore, the paragraph division includes:

[0017] S201: Calculating the height of each row according to the content of the characters arranged in rows and the font size of the characters in each row;

[0018] S202: Calculate the distance between adjacent rows according to the height of each row and the position of each row;

[0019] S203: Determine whether the spacing between each line and each adjacent line is greater than a first threshold; if the spacing between the line and one of its adjacent lines is greater than the first threshold, the line and the adjacent line are regarded as different paragraphs; if the spacing between the line and one of its adjacent lines is less than the first threshold, the line and the adjacent line are regarded as the same paragraph.

[0020] Furthermore, the paragraph division includes:

[0021] S301: Calculating the standard deviation of each adjacent row according to the content of the text arranged in rows, the font size of the text in each row, and the location of each row;

[0022] S302: Determine whether the standard deviation between each row and each adjacent row is greater than a second threshold; if the standard deviation between the row and one of its adjacent rows is greater than the second threshold, the row and the adjacent row are regarded as different paragraphs; if the standard deviation between the row and one of its adjacent rows is less than the second threshold, the row and the adjacent row are regarded as the same paragraph.

[0023] Furthermore, the vectorization described in step S105 adopts a BGE vectorization model.

[0024] A knowledge base construction system for unstructured documents, comprising:

[0025] Region division module: obtain unstructured documents and divide document regions;

[0026] Document parsing module: parsing the document area to obtain unstructured document information;

[0027] Heuristic rule processing module: using heuristic rules to process the unstructured document information to obtain an initial text;

[0028] Proofreading module: proofreading and modifying the initial text to obtain the final text;

[0029] Vectorization module: vectorize the final text to obtain a final text vector;

[0030] Knowledge base generation module: The corresponding final text vector and final text are taken as knowledge base entries and saved as a database.

[0031] A large model question answering method, comprising:

[0032] S401: Obtaining questions raised by users;

[0033] S402: vectorizing the question raised by the user to obtain a question vector;

[0034] S403: searching the database constructed by the knowledge base construction method according to the question vector to obtain N knowledge base entries with the highest similarity;

[0035] S404: The text data under the N knowledge base entries with the highest similarity and the question raised by the user are input into the big model, and the big model generates an answer.

[0036] Furthermore, the search in step S403 includes:

[0037] S501: Calculate the cosine similarity between the question vector and each final text vector in the database to obtain the cosine similarity of each final text vector;

[0038] S502: Sort each knowledge base entry according to the cosine similarity, and obtain N knowledge base entries with the largest cosine similarity as the N knowledge base entries with the highest similarity.

[0039] A large model question answering system, comprising:

[0040] Question collection module: obtain questions raised by users;

[0041] Question vectorization module: vectorizes the questions raised by users to obtain question vectors;

[0042] Knowledge base query module: retrieve the database constructed by the knowledge base construction method according to the question vector to obtain the N knowledge base entries with the highest similarity;

[0043] Answer generation module: inputs the text data under the entry and the question raised by the user into the big model, and the big model generates the answer.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention obtains unstructured documents and divides them into document areas; parses the document areas to obtain unstructured document information; processes the unstructured document information using heuristic rules to obtain initial text; proofreads and modifies the initial text to obtain final text; vectorizes the final text to obtain a final text vector; and saves the corresponding final text vector and the final text as knowledge base entries in a database; thereby, unstructured documents can be processed simply and efficiently and a knowledge base can be constructed, thereby improving the accuracy of a large language model question-answering system. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a method for constructing a knowledge base for unstructured documents provided in Example 1.

[0047] Figure 2 A flowchart of the paragraph division provided for Example 1.

[0048] Figure 3 A flowchart of the paragraph division provided for Example 1.

[0049] Figure 4 This is a structural diagram of a knowledge base construction system for unstructured documents provided in Example 1.

[0050] Figure 5 A flowchart of a large model question-answering method provided in Example 1.

[0051] Figure 6 A flowchart of the search provided in Example 1.

[0052] Figure 7 A structural diagram of a large-model question-answering system provided in Example 1. DETAILED DESCRIPTION

[0053] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0054] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0055] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0056] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0057] Example 1

[0058] like Figure 1 As shown, a method for constructing a knowledge base for unstructured documents includes:

[0059] S101: Obtain unstructured documents and divide document areas;

[0060] S102: parsing the document area to obtain unstructured document information;

[0061] S103: Processing the unstructured document information using heuristic rules to obtain an initial text;

[0062] S104: proofreading and modifying the initial text to obtain a final text;

[0063] S105: vectorizing the final text to obtain a final text vector;

[0064] S106: The corresponding final text vector and the final text are taken as knowledge base entries and saved as a database.

[0065] In a specific embodiment, the document region is parsed using pdfminer.

[0066] Furthermore, the document information includes: the content of text arranged in lines, the font size of the text in each line, and the coordinates of the location of each line and text.

[0067] Furthermore, the heuristic rules include: one or more of paragraph division, content filtering, and title recognition.

[0068] It should be noted that by using heuristic rules to process text, the initial text can be semantically segmented, thereby enhancing the semantic coherence of the initial text, so that the large model can have a better effect.

[0069] It should be noted that proofreading and modifying the initial text can enhance the correctness of the text and avoid the input of erroneous information that misleads the large model.

[0070] In a specific embodiment, matching unnecessary content in content filtering includes one or more methods of regular expression matching, exact matching, fuzzy matching, and area coordinate filtering.

[0071] In a specific embodiment, the title recognition is performed by determining whether the size of the text line is larger than a font size threshold; if it is larger than the font size threshold, it is regarded as a title.

[0072] Furthermore, if Figure 2 As shown, the paragraph division includes:

[0073] S201: Calculating the height of each row according to the content of the characters arranged in rows and the font size of the characters in each row;

[0074] S202: Calculate the distance between adjacent rows according to the height of each row and the position of each row;

[0075] S203: Determine whether the spacing between each line and each adjacent line is greater than a first threshold; if the spacing between the line and one of its adjacent lines is greater than the first threshold, the line and the adjacent line are regarded as different paragraphs; if the spacing between the line and one of its adjacent lines is less than the first threshold, the line and the adjacent line are regarded as the same paragraph.

[0076] Furthermore, if Figure 3 As shown, the paragraph division includes:

[0077] S301: Calculating the standard deviation of each adjacent row according to the content of the text arranged in rows, the font size of the text in each row, and the location of each row;

[0078] S302: Determine whether the standard deviation between each row and each adjacent row is greater than a second threshold; if the standard deviation between the row and one of its adjacent rows is greater than the second threshold, the row and the adjacent row are regarded as different paragraphs; if the standard deviation between the row and one of its adjacent rows is less than the second threshold, the row and the adjacent row are regarded as the same paragraph.

[0079] In a specific embodiment, the second threshold is 0.5.

[0080] Furthermore, the vectorization described in step S105 adopts a BGE vectorization model.

[0081] like Figure 4 As shown, a knowledge base construction system for unstructured documents includes:

[0082] Region division module: obtain unstructured documents and divide document regions;

[0083] Document parsing module: parsing the document area to obtain unstructured document information;

[0084] Heuristic rule processing module: using heuristic rules to process the unstructured document information to obtain an initial text;

[0085] Proofreading module: proofreading and modifying the initial text to obtain the final text;

[0086] Vectorization module: vectorize the final text to obtain a final text vector;

[0087] Knowledge base generation module: The corresponding final text vector and final text are taken as knowledge base entries and saved as a database.

[0088] like Figure 5 As shown, a large model question answering method includes:

[0089] S401: Obtaining questions raised by users;

[0090] S402: vectorizing the question raised by the user to obtain a question vector;

[0091] S403: searching the database constructed by the knowledge base construction method according to the question vector to obtain N knowledge base entries with the highest similarity;

[0092] S404: The text data under the N knowledge base entries with the highest similarity and the question raised by the user are input into the big model, and the big model generates an answer.

[0093] Furthermore, if Figure 6 As shown, the search in step S403 includes:

[0094] S501: Calculate the cosine similarity between the question vector and each final text vector in the database to obtain the cosine similarity of each final text vector;

[0095] S502: Sort each knowledge base entry according to the cosine similarity, and obtain N knowledge base entries with the largest cosine similarity as the N knowledge base entries with the highest similarity.

[0096] like Figure 7 As shown, a large model question answering system includes:

[0097] Question collection module: obtain questions raised by users;

[0098] Question vectorization module: vectorizes the questions raised by users to obtain question vectors;

[0099] Knowledge base query module: retrieve the database constructed by the knowledge base construction method according to the question vector to obtain the N knowledge base entries with the highest similarity;

[0100] Answer generation module: inputs the text data under the entry and the question raised by the user into the big model, and the big model generates the answer.

[0101] In a specific embodiment, a knowledge base construction interface interactive system is established to facilitate operators to better construct a knowledge base.

[0102] A knowledge base construction interface interactive system, comprising:

[0103] Workspace: Displays the document image and allows you to select a frame in the document image;

[0104] Configuration area: used to configure heuristic rules for global or specific page images;

[0105] Content extraction area: displays the results of text extraction on the current page and can also modify the current extraction results.

[0106] A knowledge base construction interface interaction method, comprising:

[0107] S601: Acquire a document, convert the document into a document image, and acquire the width and height of each page of the document and the width and height of the document image;

[0108] S602: Displaying the document image in the working area of ​​the interactive system;

[0109] S603: Acquire the area selected in the workspace, and scale it according to the width and height of each page of the document and the width and height of the document image to obtain the divided document area;

[0110] S604: Obtain the heuristic rule configuration of the current page in the configuration area, obtain the initial text, and display it in the content extraction area;

[0111] S605: Obtain the modification of the content extraction area and obtain the final text.

[0112] A method for interactively building a knowledge base interface for a table, after step S604, further comprising:

[0113] S701: Get the table area selected in the workspace frame to obtain the table area;

[0114] S702: Obtain table text according to the table area, and display it in the content extraction area.

[0115] In step S702, the table text is obtained, including: text alignment recognition: inferring the table structure by identifying the text alignment; table line recognition: constructing the row and column structure of the table by identifying the boundary lines of the table; display specified analysis: suitable for complex or irregular tables, by manually selecting the boundaries of rows and columns to accurately identify table data.

[0116] At the same time, the table text includes: table title and cell content and coordinates.

[0117] The same or similar reference numerals correspond to the same or similar components;

[0118] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0119] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for constructing a knowledge base for unstructured documents, comprising: S101: Obtain unstructured documents and divide document areas; S102: parsing the document area to obtain unstructured document information; S103: Processing the unstructured document information using heuristic rules to obtain an initial text; S104: proofreading and modifying the initial text to obtain a final text; S105: vectorizing the final text to obtain a final text vector; S106: The corresponding final text vector and the final text are taken as knowledge base entries and saved as a database.

2. A method for constructing a knowledge base for unstructured documents according to claim 1, characterized in that: The document information includes: the content of the text arranged in lines, the font size of the text in each line, and the coordinates of the location of each line and text.

3. A method for constructing a knowledge base for unstructured documents according to claim 2, characterized in that: The heuristic rules include: one or more of paragraph division, content filtering, and title recognition.

4. A method for constructing a knowledge base for unstructured documents according to claim 3, characterized in that: The paragraph division includes: S201: Calculating the height of each row according to the content of the characters arranged in rows and the font size of the characters in each row; S202: Calculate the distance between adjacent rows according to the height of each row and the position of each row; S203: Determine whether the spacing between each line and each adjacent line is greater than a first threshold; if the spacing between the line and one of its adjacent lines is greater than the first threshold, the line and the adjacent line are regarded as different paragraphs; if the spacing between the line and one of its adjacent lines is less than the first threshold, the line and the adjacent line are regarded as the same paragraph.

5. A method for constructing a knowledge base for unstructured documents according to claim 3, characterized in that: The paragraph division includes: S301: Calculating the standard deviation of each adjacent row according to the content of the text arranged in rows, the font size of the text in each row, and the location of each row; S302: Determine whether the standard deviation between each row and each adjacent row is greater than a second threshold; if the standard deviation between the row and one of its adjacent rows is greater than the second threshold, the row and the adjacent row are regarded as different paragraphs; if the standard deviation between the row and one of its adjacent rows is less than the second threshold, the row and the adjacent row are regarded as the same paragraph.

6. A method for constructing a knowledge base for unstructured documents according to claim 1, characterized in that: The vectorization described in step S105 adopts the BGE vectorization model.

7. A knowledge base construction system for unstructured documents, applied to the knowledge base construction method according to any one of claims 1 to 6, characterized in that: include: Region division module: obtain unstructured documents and divide document regions; Document parsing module: parsing the document area to obtain unstructured document information; Heuristic rule processing module: using heuristic rules to process the unstructured document information to obtain an initial text; Proofreading module: proofreading and modifying the initial text to obtain the final text; Vectorization module: vectorize the final text to obtain a final text vector; Knowledge base generation module: The corresponding final text vector and final text are taken as knowledge base entries and saved as a database.

8. A large model question answering method, characterized in that: include: S401: Obtaining questions raised by users; S402: vectorizing the question raised by the user to obtain a question vector; S403: searching the database constructed by the knowledge base construction method according to any one of claims 1 to 6 according to the question vector to obtain N knowledge base entries with the highest similarity; S404: The text data under the N knowledge base entries with the highest similarity and the question raised by the user are input into the big model, and the big model generates an answer.

9. A large model question answering method according to claim 8, characterized in that: The search in step S403 includes: S501: Calculate the cosine similarity between the question vector and each final text vector in the database to obtain the cosine similarity of each final text vector; S502: Sort each knowledge base entry according to the cosine similarity, and obtain N knowledge base entries with the largest cosine similarity as the N knowledge base entries with the highest similarity.

10. A large model question answering system, applied to the question answering method according to claim 8 or 9, characterized in that: include: Question collection module: obtain questions raised by users; Question vectorization module: vectorizes the questions raised by users to obtain question vectors; Knowledge base query module: searching the database constructed by the knowledge base construction method according to any one of claims 1 to 6 according to the question vector to obtain N knowledge base entries with the highest similarity; Answer generation module: inputs the text data under the entry and the question raised by the user into the big model, and the big model generates the answer.

Images