Large model knowledge base construction method and system based on multi-source heterogeneous file
By using a combination of Markdown format and optical character recognition in the large-scale knowledge base, the content is extracted and sliced from multi-source heterogeneous files, and a powerful knowledge base is built, which solves the problems of lack of knowledge and insufficient professionalism, and improves the great The model's Q&A effect and system performance.
Patent Information
- Application Number
- CN202411086338.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-29
AI Technical Summary
The existing large-scale model knowledge base lacks knowledge when dealing with professional fields, resulting in inaccurate and lack of professionalism, which cannot meet the specific customization requirements of vertical fields, affecting the results of question and answers.
Using a combination of Markdown format and optical character recognition, file content is extracted from multi-source heterogeneous files, and a powerful knowledge base is built through multi-scale slices and vectorization operations to meet the specific needs of vertical fields.
It improves the robustness and accuracy of the knowledge base, enhances the Q&A effect of the big model, meets the specific customization requirements of vertical fields, and improves the industry attributes and performance of the system.
Smart Images

Figure CN119047548A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of knowledge engineering, and in particular relates to a method and system for constructing a large model knowledge base based on multi-source heterogeneous files. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] 2023 is the first year of the explosion of large language models. Many manufacturers at home and abroad have released their large models, which have achieved great results in applications such as dialogue services, content creation, and intelligent search, and have attracted great attention from academia, industry, and government. The performance of large language models in the field of natural language far exceeds any previous models; however, large language models also have many shortcomings. For example, when processing knowledge in some professional fields, due to the lack of professional knowledge, they cannot provide accurate answers, and face pain points such as generating hallucinations, lack of professionalism and accuracy.
[0004] In order to solve the problem caused by knowledge deficiency, it is necessary to establish a special knowledge base for a specific field as the knowledge source of the large model to assist the large model in knowledge generation. Therefore, the construction effect of the knowledge base directly affects the answer effect of the large model.
[0005] The construction of a knowledge base includes two stages, corresponding to upstream knowledge extraction and downstream knowledge integration. In the knowledge extraction stage, it is necessary to extract as much information as possible from the knowledge (generally documents), including document content and document format; in the downstream knowledge integration stage, it is necessary to customize the construction of the knowledge base according to the current mainstream search methods.
[0006] In the process of building the knowledge base of the existing large model, industry characteristics and attributes are not taken into consideration, and the specific customization requirements of vertical field documents cannot be met, resulting in insufficient robustness and accuracy in the construction of the knowledge base, which indirectly affects the question-answering effect of the large model. Summary of the invention
[0007] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a method and system for constructing a large model knowledge base based on multi-source heterogeneous files, customizes different file content extraction strategies for multi-source heterogeneous files, and performs multi-scale slicing on the extracted content, and finally constructs a powerful and efficient knowledge base, thereby improving the large model question and answer effect.
[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:
[0009] A first aspect of the present invention provides a method for constructing a large model knowledge base based on multi-source heterogeneous files.
[0010] A method for constructing a large model knowledge base based on multi-source heterogeneous files includes:
[0011] The Markdown format and optical character recognition are combined to extract the file content including format from multi-source heterogeneous files;
[0012] According to the input data volume limit of the large model and the relevance of the topic, the extracted file content is sliced at multiple scales to obtain several file paragraphs;
[0013] Perform vectorization operations on the file paragraphs, and the paragraph vectors belonging to the same slice scale form a vector library;
[0014] The vector library of different slice scales constitutes a large model knowledge base, which is used for retrieval enhancement and generation of large models.
[0015] Furthermore, the multi-source heterogeneous files include three categories: a first category of files whose contents are text and can be directly extracted, a second category of files whose contents are text and format and can be directly extracted, and a third category of files whose contents are text and format and cannot be directly extracted.
[0016] Furthermore, the extracting of the file content including the format is specifically as follows:
[0017] For the first type of files, directly extract the file content;
[0018] For the second type of files, directly extract the file content and save it in Markdown format;
[0019] For the third type of files, a combination of table recognition and optical character recognition is used to identify the file content and save it in Markdown format.
[0020] Furthermore, the method of combining table recognition and optical character recognition to recognize the content of the file comprises the following specific steps:
[0021] Detect bordered and borderless tables in the file through the object detection algorithm;
[0022] For tables with borders, identify the borders of cells to distinguish different cells, and then perform optical character recognition on the contents of different cells;
[0023] For borderless tables, optical character recognition is performed directly, and the recognition results are post-processed to determine whether the detected text is in the same row or column through the relative position relationship of the anchor box coordinates of the detection results.
[0024] Furthermore, the multi-scale slices are specifically:
[0025] According to the length of the basic information of the large model, the length of the comprehensive prompt words, the length of the RAG reference document and the length of the context, the extracted file content is preliminarily sliced to obtain multiple file paragraphs;
[0026] Set multiple reslicing lengths, reslice continuous file paragraphs that are strongly related to the topic, and finally divide the file content into several file paragraphs.
[0027] Furthermore, the re-slicing comprises the following specific steps:
[0028] The subject of the generated file content;
[0029] Calculate the relevance of each document paragraph to the topic;
[0030] Based on the relevance, distinguish between paragraphs that are strongly related to the topic and paragraphs that are not very related to the topic;
[0031] Set multiple reslicing lengths to reslice consecutive paragraphs that are strongly related to the topic.
[0032] Furthermore, the paragraph vectors belonging to the same slice scale form a vector library, specifically:
[0033] Using the qdrant vector library as the storage vector library, after appending the payload field to the paragraph vectors belonging to the same slice scale, the paragraph vectors belonging to the same slice scale are stored in the same storage unit collection in qdrant;
[0034] The payload field stores document industry information and subject information, including industry classification, paragraph ID, content source, content source ID, pdf-ocr positioning anchor box, whether it is enabled, and article external link information.
[0035] A second aspect of the present invention provides a large model knowledge base construction system based on multi-source heterogeneous files.
[0036] A large model knowledge base construction system based on multi-source heterogeneous files, comprising:
[0037] The content extraction module is configured to: extract the file content including the format from the multi-source heterogeneous files by combining the Markdown format and optical character recognition;
[0038] The content slicing module is configured to: perform multi-scale slicing on the extracted file content according to the input data volume limit of the large model and the relevance of the topic to obtain a number of file paragraphs;
[0039] The vector library building module is configured to: perform vectorization operations on the file paragraphs, and the paragraph vectors belonging to the same slice scale form a vector library;
[0040] The knowledge base construction module is configured as follows: a vector library of different slice scales is used to form a large model knowledge base for retrieval enhancement generation of large models.
[0041] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the method for constructing a large model knowledge base based on multi-source heterogeneous files as described in the first aspect of the present invention.
[0042] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the method for constructing a large model knowledge base based on multi-source heterogeneous files as described in the first aspect of the present invention are implemented.
[0043] One or more of the above technical solutions have the following beneficial effects:
[0044] 1. The present invention adopts a set of fully functional file content extraction methods in the construction of the knowledge base, specifies different content extraction strategies for three types of files, effectively solves the specific customization requirements of vertical field documents, and improves the industry attributes and industry performance of the system.
[0045] 2. In the process of extracting text into the knowledge base, the present invention fully considers industry characteristics and industry attributes, and performs multi-scale slicing on the extracted file content according to the input data volume limit of the large model and the relevance of the topic, thereby realizing a customized high-performance vectorization process.
[0046] 3. The present invention designs a highly available and high-performance vector library architecture and constructs vector libraries of different slice scales, so as to meet the functional requirements while also meeting the performance requirements of addition, deletion, modification and query.
[0047] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0049] Figure 1 This is a flow chart of the method of the first embodiment.
[0050] Figure 2 This is a flowchart of the table format data processing of the first embodiment.
[0051] Figure 3This is a slicing flow chart of the first embodiment.
[0052] FIG. 4( a ) and FIG. 4( b ) are respectively exemplary diagrams of a table with a border and a table without a border according to the first embodiment. DETAILED DESCRIPTION
[0053] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0054] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0055] Embodiment 1
[0056] In one embodiment of the present disclosure, a method for constructing a large model knowledge base based on multi-source heterogeneous files is provided. Figure 1 As shown, the following steps are included:
[0057] Step S1: Obtain multi-source heterogeneous files;
[0058] Furthermore, the multi-source heterogeneous files include three categories:
[0059] The first type of files with text as content that can be directly extracted, such as txt text files;
[0060] The second type of files that can be directly extracted are text and formatting, such as word files;
[0061] The third category of files whose contents are text and formatting and cannot be directly extracted, such as scanned PDF files.
[0062] In industrial scenarios, there are a large number of table formats in the contents of the second and third types of files mentioned above.
[0063] Step S2: extracting the file contents including the format from the multi-source heterogeneous files by combining the Markdown format and optical character recognition;
[0064] Furthermore, the extracting of the file content including the format is specifically as follows:
[0065] For the first type of files, directly extract the file contents such as those in txt files;
[0066] For the second type of files, directly extract the file contents such as word files, including tables and other formats, and save them in Markdown format;
[0067] For the third type of files, a combination of table recognition and optical character recognition is used to identify the file content and save it in Markdown format.
[0068] Among them, Markdown is a concise, easy-to-read and easy-to-write plain text format markup language, mainly used to write easy-to-read documents. It does not do any additional operations, such as changing the font size, color or type, but it provides the basis of structure and style through a series of simple symbols and syntax, such as titles, lists, links, emphasized text, etc., and uses an easy-to-read and easy-to-write plain text format to write documents, and then converts them into valid XHTML (or HTML) documents. This language absorbs many of the features of plain text tags already in emails.
[0069] For files whose document content (i.e., containing text and formatting) cannot be extracted, such as scanned PDF files, it is necessary to combine table recognition and optical character recognition to process the text and formatting separately.
[0070] Content extraction uses the traditional optical text recognition OCR method, which has the advantages of high versatility, high speed, and good recognition effect on printed text. It can quickly extract text content from PDF. This is the current mainstream processing for scanned PDF files, but this method is less effective for extracting table content from scanned PDF files.
[0071] This embodiment performs targeted processing on the table format data in the scanned PDF document. The processing method flow is as follows: Figure 2 As shown, the specific steps are:
[0072] Step S2-01: Detect bordered tables and borderless tables in the file through a target detection algorithm;
[0073] Specifically, when detecting the table part in the scanned PDF file, you can choose the yolo v5s target detection algorithm to implement the target detection of the table, including the bordered table shown in Figure 4(a) and the borderless table shown in Figure 4(b). The detection difficulty of the bordered table is relatively low, while the detection difficulty of the borderless table is relatively high.
[0074] In the data annotation process for model training, it is necessary to appropriately expand the scope of the anchor box and use the blank space between the borderless table and the text to assist in table detection.
[0075] In the actual process, the bordered table and the borderless table need to be detected separately. The bordered table is detected first. If the document content of the bordered table is not detected, the borderless table is detected.
[0076] When using the yolo v5s target detection algorithm to detect bordered tables, the detection object is the cell separator, that is, the horizontal or vertical solid line, specifically:
[0077] (1) Convert each page of the document PDF into an image format and send it to the detection model built based on YOLO v5s. The model detects all similar solid lines in the document. While detecting the table border, it also detects other dividing lines or text box borders.
[0078] (2) Perform business processing based on the detected solid line:
[0079] The number of vertical solid lines between two horizontal solid lines is used to distinguish between tables, other text boxes, or separators. For example, if there are more than two vertical solid lines between two horizontal solid lines, the content between the two horizontal lines is considered to be in table format.
[0080] When using the yolo v5s target detection algorithm to detect borderless tables, since the detection object is a text structure different from conventional typesetting, there is no solid line to separate the boundary between the recognized table and the normal format content. Therefore, the feature used to detect the table boundary is a blank area that is much larger than the line spacing.
[0081] Step S2-02: For a table with borders, identify the borders of cells to distinguish different cells, and then perform optical character recognition on the contents of different cells;
[0082] Step S2-03: For borderless tables, directly perform optical character recognition, and post-process the recognition results to determine whether the detected text is in the same row or column by detecting the relative position relationship of the anchor frame coordinates. Specifically:
[0083] Assuming that two anchor boxes are in the same row, there should be a certain interval between the horizontal coordinates of the two anchor boxes. This interval is related to the format of the text and is fixed. Therefore, when the interval between the horizontal coordinates of the two anchor boxes meets a certain threshold requirement, they are considered to be in the same row.
[0084] Similar to the way of judging the same line, when the interval between the vertical coordinates of two anchor boxes meets a certain threshold requirement, they are considered to be in the same column.
[0085] In steps S2-02 and S2-03, there are problems such as "the number of rows in the table in the PDF file is too long" and "the number of columns in the table in the PDF file is too many". These two types of problems are handled as follows:
[0086] (1) The problem of too many rows in the table in the PDF file:
[0087] The content of the table exceeds the maximum length of the large model input sequence, resulting in the inability to cover the header information when slicing, causing the large model to have insufficient understanding of the physical meaning of each column cell, affecting the correctness of the large model output results.
[0088] This embodiment aims at the cross-page table scenario. When slicing the table, the header of each table is extracted and added to each slice with table content as a header as additional reference information. In this way, when the large model performs knowledge retrieval, it can refer to the header information when retrieving each slice and understand the physical meaning of each cell. In terms of the process, after completing the recognition of the table, the table content enhancement step is first entered, and data enhancement of long tables is given priority.
[0089] (2) Problem of too many columns in the table in PDF file:
[0090] It is difficult for table recognition technology to capture the logical relationship between fields in each column of the same row of cells, resulting in better question-and-answer results when asking questions about adjacent columns of a table; and poorer results when asking questions about columns with larger intervals between them.
[0091] When faced with a wide table, the present invention not only stores the table content into the knowledge base in Markdown format, but also uses unstructured data description to strengthen the connection between columns with large intervals. For example, assuming that name, height, and weight are columns with large intervals, the information after processing through unstructured description is as follows: Xiao Ming's height is 180cm, and Xiao Ming's weight is 80KG; in terms of process, the wide table is optimized after the long table is optimized.
[0092] Step S3: According to the input data volume limit of the large model and the relevance of the topic, the extracted file content is sliced at multiple scales, and each paragraph is obtained by vectorizing the slices;
[0093] Furthermore, the multi-scale slices are specifically:
[0094] According to the length of the basic information of the large model, the length of the comprehensive prompt words, the length of the RAG reference document and the length of the context, the initial slicing length is selected, and the extracted file content is initially sliced to obtain multiple file paragraphs;
[0095] Set multiple reslicing lengths, reslice continuous file paragraphs that are strongly related to the topic, and finally divide the file content into several file paragraphs.
[0096] Specifically, when text is vectorized, different slicing strategies are formulated for different types of text and text with different retrieval requirements. The process is as follows:
[0097] Step S3-01: According to the length of the basic information of the large model, the length of the comprehensive prompt words, the length of the RAG reference document and the length of the context, the initial slice length is selected. The formula is as follows:
[0098] L = embedding(Lh+L prompt+ ∑L content+ ∑L seq )
[0099] Among them, L h It is the basic information length of the large model, namely the identity definition of the question-answering assistant, how to respond to uncertain answers, its own identity information (such as whether the model is developed by a certain company), and the length of sentences used to improve the human-computer interaction experience.
[0100] L prompt It is the length of the comprehensive prompt words, that is, the total number of words after the special prompt word (prompt) is evaluated and combined with the target question. It is the sentence length after the special prompt formulated for this question and the question filled in are combined.
[0101] ∑L content It is the length of RAG reference document, that is, the total number of words in the historical or related documents used to prompt the model in the retrieval-assisted generation scenario. It is the total length of the text in RAG as the large model prompt paragraph. For example, if the maximum length of each slice is 500, and the system uses the 5 slices most relevant to the question as reference knowledge, it will take up 5*500=2500 words.
[0102] ∑L seq It is the context length, that is, the total amount of context information accumulated in the continuous dialogue. It is the total length of context sentences in multi-round dialogues supported by the large model. For example, if the number of rounds of multi-round dialogues supported is 3, and the estimated sentence length of each answer plus the question is 300, then 300*3=900 characters are required.
[0103] After the addition, the enbedding model is used for encoding to convert the length of the Chinese sentence into the number of tokens. The text content of the sentence will be encoded, and some special identifiers will be added as markers for the start and end of the sentence. The sum of the common token numbers of the two parts is the total required token number overhead, which needs to be less than the maximum token number of the base large model.
[0104] In the above variables, ∑L seq The size of is the most difficult to estimate. If the maximum length of the slice is set improperly, combined with ∑L seq Sudden changes may cause video memory overflow, so when setting the maximum length of the slice, you need to leave enough margin to prevent the total number of tokens from exceeding the limit of LLM.
[0105] Step S3-02: Based on the above rules, the maximum length of a single slice is selected as the preliminary slice length, and 500 is taken as an example below.
[0106] When slicing a document, you need to pay attention not only to the maximum length, but also to set a reasonable overlap length. That is, in order to make the semantics of a single slice more complete and better preserve the contextual relationship, adjacent slices will have a small amount of text overlap. The text overlap length should not be too large. The following takes 20 as an example.
[0107] Step S3-03: Based on the above-mentioned slice length and overlap length, corresponding slice strategies need to be formulated for different types of special texts, which will help improve the accuracy of subsequent knowledge retrieval, specifically:
[0108] (1) Slicing strategy for general format content
[0109] When facing Markdown type documents, you should try to ensure that the contents of the same level of headings exist in the same slice when slicing. This can minimize the loss of information caused by the separation of the original content due to document slicing. If the contents of the same level of headings cannot be completely placed in the same slice, you should try to ensure that the same sentence and paragraph are in the same slice when slicing. The mark to distinguish whether it is the same sentence or paragraph is special symbols, such as commas, line breaks, etc.
[0110] When setting the text overlap length, since this parameter is generally small and can hardly cover a paragraph or sentence, the mainstream solution is to set the overlapped text content length directly according to the maximum length, which is easy to cause gradient disappearance or gradient explosion problems. This embodiment adopts a dynamic setting method:
[0111] If the overlap length is too short to cover the sentence of the previous slice, the title corresponding to the slice in the original text will be used as context information and inserted into the content of the slice as a summary of the context information. The remaining overlap margin will be set according to the maximum length.
[0112] (2) Facing the re-slicing strategy with table format
[0113] The mainstream document processing process in the industry does not make any special processing for the table format. It is just expressed in the table format Markdown and divided as part of the content. This embodiment makes special processing for the table data:
[0114] As a kind of heterogeneous data, table data needs to be processed separately from text. Considering that the data in each row of the table is almost relatively independent, except for special cases, the data information in each row fully reflects the real situation; when slicing, this embodiment uses the line break of the table as the division standard. This method retains the information of the same row in the table in the same slice to the greatest extent possible when slicing, minimizes the loss of information, and makes the question and answer results more accurate.
[0115] (3) Re-slicing strategy for important documents
[0116] In the mainstream document processing in the industry, the slice size is fixed, while the distribution of important information in the document is uneven, that is, some paragraphs contain important information related to the topic, while other paragraphs are not closely related to the topic. In the face of this situation, according to the mainstream document processing method, that is, the method of segmenting the document with the same slice size, the question-answering effect is not good. To this end, this embodiment provides the following Figure 3 The slicing process shown:
[0117] 1) First, the topic of the full text is generated based on the capabilities of the big model. In this step, customized prompt words and colloquial instructions need to be input into the big model, such as "Please summarize the topic of the above text for me", etc. The document and the prompt words are used as the input of the model. At this time, the big model will output the topic of the full text; this result will be used to detect the relevance of each slice and topic in the subsequent process.
[0118] 2) Perform preliminary slicing and vectorization on the document according to the slicing length set in step S3-02 to obtain a preliminary slicing vector library.
[0119] 3) After the user enters the question, the user question needs to be vectorized. Using the user question as a reference, the preliminary slice vector library is vectorized and searched through cosine similarity. The formula is as follows:
[0120] Cosine similarity
[0121] Among them, A is the vectorized user question, B is the slice vector in the preliminary slice vector library, and the calculated cosine similarity is used as the similarity of the two vectors. The similarity between each slice vector in the preliminary slice vector library and the user question is calculated in the above way, and then they are arranged in descending order, and the highest K vectors are taken, that is, they are arranged from high to low in terms of vector similarity with the vectorized user question by reordering.
[0122] 4) Based on similarity and re-ranking, a connectivity rule judgment algorithm based on relevance is proposed to judge the distribution of important information in the document, which is mainly divided into the following steps:
[0123] The first step is to use the capabilities of the big model to calculate the relevance score between each slice and the topic. This step requires the formulation of special prompt words, such as "Please determine the relevance between the following document and XXX, and score it. The score range is 1-100". Replace XXX in the above text with the slice topic, and use it together with the slice content as the input parameter of the big model. At this time, the big model will output a score between 1-100 to evaluate the relevance between the slice and the topic.
[0124] The second step is to develop a relevance screening standard to distinguish slices that are strongly related to the document topic from slices that are not closely related to the document topic. Slices above the threshold are called positive examples, and slices below the threshold are called negative examples.
[0125] The third step is to arrange the slices according to the order of the source document and make the following judgments:
[0126] If there are no two or more adjacent slices that are positive examples, the important information is evenly distributed in the document and there is no need to re-slice.
[0127] If there are two or more adjacent slices that are positive examples, the distribution of important information in the document is uneven and needs to be re-sliced. Then these adjacent slices are selected until negative examples appear and these adjacent positive slices are recombined in the order of the original document to form a sub-document; after this stage, the original document becomes several sub-documents that are strongly related to the topic.
[0128] The fourth step is to re-slice the obtained sub-documents, and re-slice twice using two re-slice lengths, one of which is longer than the initial slice length, and the other is shorter than the initial slice length. Taking the storage time slice length setting value of 500 as an example, the two re-slice lengths are 300 and 700 respectively. The two re-slices and the original slices (i.e., sub-documents) are stored in the vector library together as part of the knowledge base, that is, the paragraphs after re-slicing are stored in the knowledge base together with the original paragraphs.
[0129] Compared with the mainstream knowledge construction method, the method of this embodiment effectively alleviates the problem of source document information loss caused by unreasonable slice length setting, and retains the meaning of the original document to the greatest extent; at the same time, it utilizes the capabilities of the large model to minimize the length of the re-sliced text and reduce the resource overhead of the entire system.
[0130] Step S4: Paragraph vectors belonging to the same slice scale form a vector library, and vector libraries of different slice scales constitute a large model knowledge base, which is used for retrieval enhancement generation of the large model.
[0131] Furthermore, the vector library uses the qdrant vector library as the storage vector library, and adds a payload field to paragraphs belonging to the same slice scale. The payload field contains document industry information and subject information, including industry classification, paragraph ID, content source, content source ID, pdf-ocr positioning anchor box, whether to enable, and article external link information.
[0132] In terms of data design and search index setting of the vector library, this embodiment is based on actual industrial scenarios and comprehensively considers factors such as construction cost, vector library performance, architecture scalability, and ease of use.
[0133] (1) Vector library selection
[0134] This embodiment is based on the data characteristics of the industrial industry, that is, there are many sub-industries, high professional barriers between industries and low knowledge sharing. Even for the same sub-industry, different factories will have different "specific rules". The data of each industry includes general and private parts. The cross-overlap between data is relatively high, and the search scope involved is complex; precisely because the data is divided in detail, the total amount of data of each factory is not particularly high, and not much knowledge can enter the general knowledge base.
[0135] This embodiment uses qdrant as the database, whose largest storage unit is collection, corresponding to the database in the relational database. The method of this embodiment is different from other mainstream methods in that the criterion for dividing the collection is the embedding model mentioned above or the difference in the slice length split_size, and does not include the data isolation of different industries in business logic. In this embodiment, the data isolation of different industries in business logic is distinguished by the payload field. The advantages of this method over the traditional business logic-based database and table division are:
[0136] First, the data division standard at the upper layer of business logic, namely different specifications of text vectorization, is well integrated into the overall design system;
[0137] Second, based on the characteristics of different queries, under the same vectorization standard, the query scope divided by business logic is diverse. If strong data isolation at the collection level is added, a large number of cross-collection queries will appear under the diversified search scope, and the query efficiency is poor. Data between different standards (that is, data at different slice scales) will not be queried together at all, and the queries are independent of each other, so the strongest data isolation type collection is selected.
[0138] (2)Payload field selection
[0139] As an important means to distinguish industries and topics, payload needs to contain rich resources other than text content.
[0140] The payload in this embodiment stores the industry, paragraph ID, content source, content source ID, PDF-OCR anchor box, whether to enable, article external links, etc. Unique IDs are assigned to paragraphs and content sources. The IDs are generated based on the file name and paragraph content. The content source IDs are consistent in different collections, which facilitates searches for the same document with different vectorization standards. At the same time, the generated unique content source ID can also be used as a field for building efficient indexes to optimize searches.
[0141] When searching the vector library built based on the above method, at the beginning of the search, a similarity search of the initial vectorization parameter collection will be performed first. After the search results are rearranged, relevant paragraphs are searched in collections of different vectorization standards according to different document IDs, and then integrated; they will be used as references for large-model question and answering.
[0142] When using the SDK to connect to the vector library, the client needs to be instantiated, and the same client can only connect to the same collection. Therefore, all the data of this patent is stored in one collection under the same slicing standard, and metadata is used to distinguish different industry fields. Metadata is the object storage function of qdrant, which can store structured or unstructured data. Use a field in metadata to distinguish different domain knowledge, and add indexes to the fields to ensure query efficiency.
[0143] Embodiment 2
[0144] In one embodiment of the present disclosure, a large model knowledge base construction system based on multi-source heterogeneous files is provided, including:
[0145] The content extraction module is configured to: extract the file content including the format from the multi-source heterogeneous files by combining the Markdown format and optical character recognition;
[0146] The content slicing module is configured to: perform multi-scale slicing on the extracted file content according to the input data volume limit of the large model and the relevance of the topic to obtain a number of file paragraphs;
[0147] The vector library building module is configured to: perform vectorization operations on the file paragraphs, and the paragraph vectors belonging to the same slice scale form a vector library;
[0148] The knowledge base construction module is configured as follows: a vector library of different slice scales is used to form a large model knowledge base for retrieval enhancement generation of large models.
[0149] Embodiment 3
[0150] The purpose of this embodiment is to provide a computer-readable storage medium.
[0151] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in a method for constructing a large model knowledge base based on multi-source heterogeneous files as described in the first embodiment of the present disclosure.
[0152] Embodiment 4
[0153] The purpose of this embodiment is to provide an electronic device.
[0154] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in a method for constructing a large model knowledge base based on multi-source heterogeneous files as described in the first embodiment of the present disclosure are implemented.
[0155] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for constructing a large model knowledge base based on multi-source heterogeneous files, characterized in that: include: The Markdown format and optical character recognition are combined to extract the file content including format from multi-source heterogeneous files; According to the input data volume limit of the large model and the relevance of the topic, the extracted file content is sliced at multiple scales to obtain several file paragraphs; Perform vectorization operations on the file paragraphs, and the paragraph vectors belonging to the same slice scale form a vector library; The vector library of different slice scales constitutes a large model knowledge base, which is used for retrieval enhancement and generation of large models.
2. A method for constructing a large model knowledge base based on multi-source heterogeneous files as claimed in claim 1, characterized in that: The multi-source heterogeneous files include three categories: a first category of files whose contents are text and can be directly extracted, a second category of files whose contents are text and format and can be directly extracted, and a third category of files whose contents are text and format and cannot be directly extracted.
3. A method for constructing a large model knowledge base based on multi-source heterogeneous files as claimed in claim 1, characterized in that: The extraction includes the file contents in the format, specifically: For the first type of files, directly extract the file content; For the second type of files, directly extract the file content and save it in Markdown format; For the third type of files, a combination of table recognition and optical character recognition is used to identify the file content and save it in Markdown format.
4. A method for constructing a large model knowledge base based on multi-source heterogeneous files as claimed in claim 3, characterized in that: The method of combining table recognition and optical character recognition to recognize the content of the file is specifically performed as follows: Detect bordered and borderless tables in the file through the object detection algorithm; For tables with borders, identify the borders of cells to distinguish different cells, and then perform optical character recognition on the contents of different cells; For borderless tables, optical character recognition is performed directly, and the recognition results are post-processed to determine whether the detected text is in the same row or column through the relative position relationship of the anchor box coordinates of the detection results.
5. The method for constructing a large model knowledge base based on multi-source heterogeneous files as claimed in claim 1, characterized in that: The multi-scale slices are specifically: According to the length of the basic information of the large model, the length of the comprehensive prompt words, the length of the RAG reference document and the length of the context, the initial slicing length is selected, and the extracted file content is initially sliced to obtain multiple file paragraphs; Set multiple reslicing lengths, reslice continuous file paragraphs that are strongly related to the topic, and finally divide the file content into several file paragraphs.
6. A method for constructing a large model knowledge base based on multi-source heterogeneous files as claimed in claim 5, characterized in that: The re-slicing comprises the following specific steps: The subject of the generated file content; Calculate the relevance of each document paragraph to the topic; Based on the relevance, distinguish between paragraphs that are strongly related to the topic and paragraphs that are not very related to the topic; Set multiple reslicing lengths to reslice consecutive paragraphs that are strongly related to the topic.
7. A method for constructing a large model knowledge base based on multi-source heterogeneous files as claimed in claim 1, characterized in that: The paragraph vectors belonging to the same slice scale form a vector library, specifically: Using the qdrant vector library as the storage vector library, after appending the payload field to the paragraph vectors belonging to the same slice scale, the paragraph vectors belonging to the same slice scale are stored in the same storage unit collection in qdrant; The payload field stores document industry information and subject information, including industry classification, paragraph ID, content source, content source ID, pdf-ocr positioning anchor box, whether it is enabled, and article external link information.
8. A large model knowledge base construction system based on multi-source heterogeneous files, characterized in that: include: The content extraction module is configured to: extract the file content including the format from the multi-source heterogeneous files by combining the Markdown format and optical character recognition; The content slicing module is configured to: perform multi-scale slicing on the extracted file content according to the input data volume limit of the large model and the relevance of the topic to obtain a number of file paragraphs; The vector library building module is configured to: perform vectorization operations on the file paragraphs, and the paragraph vectors belonging to the same slice scale form a vector library; The knowledge base construction module is configured as follows: a vector library of different slice scales is used to form a large model knowledge base for retrieval enhancement generation of large models.
9. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; a processor for executing the computer-readable instructions; Wherein, when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is executed.
10. A storage medium, characterized in that: The computer-readable instructions are non-transitorily stored, wherein when the computer-readable instructions are executed by a computer, the method of any one of claims 1 to 7 is performed.
Citation Information
Cited By
Knowledge base document intention recognition method based on layout recognition
CN121072695A
