Knowledge base construction method and system of large model

Abstract

The invention discloses a knowledge base construction method and system for a large model, and relates to the technical field of information, and the method comprises the steps: preprocessing a multi-source heterogeneous file, and extracting and standardizing the content; the text is sliced and vectorized, and context information is reserved; and storing the vectorized data into a vector library, and executing indexing and optimization. According to the knowledge base construction method of the large model, a set of document extraction method with complete functions is adopted in knowledge base construction, the specific customization requirement of vertical field documents is effectively met, and the industry attribute and the industry performance of a system are improved; industry characteristics and industry attributes are fully considered in the process of entering the extracted text into the knowledge base, and a customized high-performance vectorization process is provided; and a vector library architecture with high availability and high performance is designed, so that the function requirements are met, and the performance requirements of addition, deletion, modification and checking are also met.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method and system for constructing a knowledge base of a large model. Background Art

[0002] In 2023, the big language model exploded. Whether it is the GPT series model produced by CloseAI, the LLAma series open source model, Google's Gemini and other models, their performance is amazing. The performance of the big language model in the field of natural language far exceeds any previous model. However, the big language model also has many shortcomings: when dealing with knowledge in some professional fields, it will show knowledge loss, and the big language model may not be able to provide accurate answers at this time. When solving such problems, data scientists usually use the method of fine-tuning the model to adapt to the knowledge of a specific field and parameterizing the knowledge. Although this method has achieved excellent results, its disadvantage is that it is costly and requires professional technical knowledge.

[0003] Another solution for large language models: parameterized knowledge (fine-tuning) has great limitations. It is difficult to retain all the knowledge in the training corpus. Each update of knowledge consumes a lot of computing resources to train the model. Model parameters cannot be updated dynamically, and parameterized knowledge can become outdated at any time. However, compared to parameterized knowledge (i.e., adapting to professional knowledge through model fine-tuning), non-parametric knowledge, that is, knowledge sources stored externally. It is more convenient and easy to expand. This approach eliminates the need for developers to retrain the entire huge model for each specific task. They can simply add a knowledge base to the model, which increases the information input of the model and thus improves the accuracy of the answer. In order to combine the advantages and disadvantages of the two methods, the model can take a semi-parametric approach, combining a non-parametric corpus database with a parameterized model. This method is called retrieval-enhanced generation.

[0004] In order to enhance retrieval, a huge data knowledge base is needed as a foundation. How to build the knowledge base and search it is the main problem currently faced. Summary of the invention

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by the present invention is: the existing large-model knowledge base construction method has the problems of insufficient efficiency in processing multi-source heterogeneous data, insufficient slicing and vectorization accuracy, and low vector library retrieval efficiency, as well as how to improve the efficiency of knowledge base construction and optimization of retrieval accuracy.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: A method for constructing a knowledge base of a large model, comprising:

[0008] Preprocess multi-source heterogeneous files, extract and standardize content;

[0009] Slice and vectorize text to preserve contextual information;

[0010] Store vectorized data into the vector library and perform indexing and optimization.

[0011] As a preferred solution of the method for constructing a knowledge base of a large model described in the present invention, wherein: the multi-source heterogeneous files include structured data files, semi-structured data files and unstructured data files;

[0012] The extracting and standardizing the content includes directly extracting text content from the structured data file;

[0013] Extract text content from semi-structured data files and convert format information into a unified markup format;

[0014] Use optical character recognition technology to extract text from unstructured data files and parse and process special format content in documents.

[0015] As a preferred solution of the knowledge base construction method of the large model described in the present invention, wherein: the special format content in the document includes a detection table area, and the target detection model is used to identify the table with borders and the table without borders;

[0016] For bordered tables, identify cell borders and perform OCR processing;

[0017] For borderless tables, use OCR recognition directly to determine the row and column relationship of the text through the anchor box coordinates;

[0018] For table data that spans pages or slices, extract the header information and keep the header in the slice;

[0019] Optimize wide tables and strengthen relationships between distant columns through unstructured descriptions.

[0020] As a preferred solution of the knowledge base construction method of the large model described in the present invention, the text slicing and vectorizing includes preliminarily determining the set slice length according to the maximum number of input tokens of the base large model, and formulating corresponding slicing strategies for different types of special texts.

[0021] As a preferred solution of the method for constructing a knowledge base of a large model described in the present invention, wherein: the slice length is expressed as,

[0022] L=encding(L h +L prompt +∑L content +Lseq )

[0023] Among them, L h is the basic information of the large model, L prompt is the length of the sentence after the special prompt formulated for this question and the filled-in question is combined, ∑L connant is the total length of the text in the RAG as a large model prompt paragraph, ∑L seq is the total length of context sentences in multi-round conversations supported by the large model, and encding represents the encding model.

[0024] As a preferred solution of the knowledge base construction method of the large model described in the present invention, the slicing strategy includes, for Markdown type documents, giving priority to putting the contents under the same level title into the same slice when slicing;

[0025] If it cannot be accommodated completely, keep the sentence or paragraph in the same slice;

[0026] The length of the overlapping part of the slices is dynamically adjusted according to the semantic structure of the text. If the overlapping part is not enough to cover the sentence of the previous slice, the original title information is inserted into the slice content.

[0027] As a preferred solution of the knowledge base construction method of the large model described in the present invention, wherein: the vectorized data is stored in the vector library, and the indexing and optimization are performed, including selecting a vector database based on the data characteristics of the industrial field, whose minimum storage unit is a collection, dividing different collections according to the text vectorization model or the slice size, and distinguishing the data through the payload field;

[0028] Select the payload field to distinguish industries and topics. The fields include industry, paragraph ID, content source ID, and create indexes for the fields.

[0029] Use hot and cold separation technology to store frequently accessed data segments in hot storage, and infrequently accessed data in cold storage. When searching, search in hot storage first. If the demand cannot be met, turn to cold storage.

[0030] During the search process, a similarity search of the initial vectorized parameters is first performed. After the search results are rearranged, relevant paragraphs are retrieved from different collections based on the document ID. After the search is completed, the final answer output is generated by integrating the various parts of the content. All data is stored in the same collection, and different industries and domain knowledge are distinguished through metadata fields.

[0031] Another object of the present invention is to provide a large-model knowledge base construction system, which can solve the problems of inconsistent multi-source heterogeneous data processing, inaccurate retrieval after vectorization, and slow retrieval speed in existing knowledge base construction methods by constructing a large-model knowledge base construction system.

[0032] In order to solve the above technical problems, the present invention provides the following technical solutions: a large model knowledge base construction system, comprising: a text preprocessing module, a text slicing module and a knowledge base construction module;

[0033] The text preprocessing module is used to preprocess multi-source heterogeneous files and extract and standardize content;

[0034] The text slicing module is used to slice and vectorize the text to retain context information;

[0035] The knowledge base construction module is used to store vectorized data into a vector library and perform indexing and optimization.

[0036] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method for constructing a knowledge base of a large model as described above when executing the computer program.

[0037] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for constructing a knowledge base of a large model as described above.

[0038] Beneficial effects of the present invention: The large-model knowledge base construction method provided by the present invention adopts a set of fully functional document extraction methods in the construction of the knowledge base, which effectively solves the specific customization requirements of vertical field documents and improves the industry attributes and industry performance of the system;

[0039] When the extracted text enters the knowledge base, the industry characteristics and attributes are fully considered, and there is a customized high-performance vectorization process;

[0040] Design a highly available and high-performance vector library architecture to meet the functional requirements while also meeting the performance requirements of adding, deleting, modifying and querying. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0042] Figure 1An overall flow chart of a method for constructing a knowledge base of a large model provided by an embodiment of the present invention.

[0043] Figure 2 A flowchart of PDF data enhancement of a special type (with tables) in a method for building a knowledge base of a large model provided by an embodiment of the present invention.

[0044] Figure 3 A logic flow chart of hot and cold knowledge base searches for a large model knowledge base construction method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0046] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0047] Example 1, reference Figure 1 , as an embodiment of the present invention, provides a method for constructing a knowledge base of a large model, comprising:

[0048] Preprocess multi-source heterogeneous files, extract and standardize content;

[0049] Slice and vectorize text to preserve contextual information;

[0050] Store vectorized data into the vector library and perform indexing and optimization.

[0051] The multi-source heterogeneous files include structured data files, semi-structured data files and unstructured data files;

[0052] The extracting and standardizing the content includes directly extracting text content from the structured data file;

[0053] Extract text content from semi-structured data files and convert format information into a unified markup format;

[0054] Use optical character recognition technology to extract text from unstructured data files and parse and process special format content in documents.

[0055] The special format content in the document includes detecting table areas, and using an object detection model to identify tables with and without borders;

[0056] For bordered tables, identify cell borders and perform OCR processing;

[0057] For borderless tables, use OCR recognition directly to determine the row and column relationship of the text through the anchor box coordinates;

[0058] For table data that spans pages or slices, extract the header information and keep the header in the slice;

[0059] Optimize wide tables and strengthen relationships between distant columns through unstructured descriptions.

[0060] The text slicing and vectorizing includes preliminarily determining the set slice length according to the maximum number of input tokens of the base large model, and formulating corresponding slicing strategies for different types of special texts.

[0061] The slice length is expressed as,

[0062] L=encding(L h +L prompt +∑L content +L seq )

[0063] Among them, L h is the basic information of the large model, L prompt is the length of the sentence after the special prompt formulated for this question and the filled-in question is combined, ∑L connant is the total length of the text in the RAG as a large model prompt paragraph, ∑L seq is the total length of context sentences in multi-round conversations supported by the large model, and encding represents the encding model.

[0064] The slicing strategy includes, for Markdown documents, giving priority to putting the contents under the same level heading into the same slice when slicing;

[0065] If it cannot be accommodated completely, keep the sentence or paragraph in the same slice;

[0066] The length of the overlapping part of the slices is dynamically adjusted according to the semantic structure of the text. If the overlapping part is not enough to cover the sentence of the previous slice, the original title information is inserted into the slice content.

[0067] The storing of vectorized data into a vector library, performing indexing and optimization includes selecting a vector database based on data characteristics in the industrial field, wherein the minimum storage unit is a collection, dividing different collections according to a text vectorization model or a slice size, and distinguishing data through a payload field;

[0068] Select the payload field to distinguish industries and topics. The fields include industry, paragraph ID, content source ID, and create indexes for the fields.

[0069] Use hot and cold separation technology to store frequently accessed data segments in hot storage, and infrequently accessed data in cold storage. When searching, search in hot storage first. If the demand cannot be met, turn to cold storage.

[0070] During the search process, a similarity search of the initial vectorized parameters is first performed. After the search results are rearranged, relevant paragraphs are retrieved from different collections based on the document ID. After the search is completed, the final answer output is generated by integrating the various parts of the content. All data is stored in the same collection, and different industries and domain knowledge are distinguished through metadata fields.

[0071] Example 2, reference Figure 2-Figure 3 , is an embodiment of the present invention, in order to solve the following problems and problem examples that occur during the construction process:

[0072] 1. Many types of documents

[0073] In real scenarios, there are multi-source heterogeneous file types, such as doc, docx, ppt, pdf, etc. Different file encoding formats are different, and the information contained is also different. For example, the pdf format has different flags for content in different formats, such as tables, pictures, etc. When facing doc / docx type files, you need to use Microsoft's deserialization tool to crack them, which makes data integration difficult.

[0074] 2. Slice size is difficult to determine

[0075] The document cannot be input into the large model in one go and needs to be sliced ​​and then vectorized. The length of the slice will have a great impact on the question and answer of the large model. If it is too long, too much hardware resources will be consumed. If it is too short, it will not be able to fully express the original meaning including the context, resulting in a reduction in the accuracy of question and answer.

[0076] 3. Disadvantages of vector retrieval

[0077] 3.1 Accuracy of vector library search

[0078] If the query is too short, such as only an ID, a hash code, or a product name, then their word vectors may not reflect their true meaning, and cannot be effectively matched with other related documents. This will lead to inaccurate vector retrieval results, or even some completely irrelevant content. Similarly, if we query for keywords such as "8XLARGE64" and "99.9%", the vector search will produce irrelevant content, making the large model behind it useless and even misleading.

[0079] 3.2 Speed ​​of vector library search

[0080] As more and more knowledge is accumulated, the search of vectors will also become slower and slower. The traditional method of dividing the knowledge base according to business logic can alleviate some problems, but it ignores the coherence and relevance between knowledge systems. In real scenarios, the effect of dividing business logic according to industry and profession is that the accuracy is greatly reduced while the speed is guaranteed. How to achieve a balance between speed and accuracy is also an issue that needs attention.

[0081] A method for constructing a knowledge base of a large model is provided, comprising:

[0082] 1) Processing of multi-source heterogeneous documents during knowledge extraction and construction;

[0083] 1.1) For files whose contents can be directly extracted, just extract the text directly

[0084] 1.2) For files that can directly extract text and formatting, in order to preserve the format and content of the original text as much as possible, convert the source text into a special format, such as the most widely used Markdown format on the market. The Markdown format uses special characters that are not used in the main text to identify the overall format. In the content slicing process, the entire document can be divided according to different heading levels (such as first-level headings, second-level headings), try to ensure the integrity of the selected heading level content, and put the content of the same level heading in the same slice as much as possible.

[0085] 1.3) For files that cannot extract documents and formats, such as scanned PDFs, the content and format need to be processed separately. The content is extracted using the traditional optical-based text recognition OCR method. The advantages are high versatility and high speed, good recognition effect on printed text, and the ability to quickly extract text content from PDF. This is also the mainstream processing for scanned PDF files. In practice, scanned PDF files generally contain special formats such as tables and drawings. However, since the base large model of this method is a language-based large model, it is unable to process multimodal data such as images and is not considered. In this system, targeted processing is performed on the scanned table data, and the overall processing flow is as follows: Figure 2 As shown, after the content in a special format is processed specifically, traditional OCR is performed. Figure 2 ss1.5 in.

[0086] 1.4) Perform the following two steps for the tables that appear in the scanned documents: a. Table detection, which parts of the PDF file need to be detected as tables, and the target detection of the table is performed based on yolov5s through target detection, including tables with borders and tables without borders. It is easier to detect tables with borders, and more difficult to detect tables without borders. In the process of data annotation for model training, it is necessary to appropriately expand the range of the anchor box, and use the blank space between the borderless table and the text to assist in the detection of the table. In the actual process, tables with borders and tables without borders need to be detected separately. Taking the detection of tables with borders as an example, Figure 2 In ss1.1, the document content of the table is not detected and the borderless table is detected, such as Figure 2 b. Table recognition: For bordered tables, a special detection model is used to identify the borders of cells to distinguish different cells, and then OCR recognition is performed on the contents of different cells; for borderless tables, OCR is directly used for recognition, and the recognition results are post-processed to determine whether the detected text is in the same row or column through the relative position relationship of the anchor box coordinates of the detection results.

[0087] 1.5) For all the table data in PDF documents, many tables have content that is too long and exceeds the maximum sequence length of the large model input. When slicing, the header information cannot be included, which will cause the large model to have insufficient understanding of the physical meaning of each column cell, affecting the correctness of the large model output results. When processing cross-page and cross-slice tables, this system extracts the header of each table and adds it as additional reference information to each slice with a table as a header. In this way, when the large model performs knowledge retrieval, it can refer to the header information when retrieving each slice and understand the physical meaning of each cell. In terms of the process, after the table is recognized, the table content enhancement step begins, see Figure 2 In ss1.3, data enhancement for long tables is prioritized.

[0088] 1.6) For the case where the table is too wide. It is difficult for table recognition technology to capture the logical relationship between the fields in each column of the same row of cells, which will result in better question-answering results for questions about adjacent columns of the table; and poor question-answering results for columns with large intervals. When facing wide tables, this system not only stores the table content into the knowledge base in markdown format, but also uses unstructured data descriptions to strengthen the connection between columns with large intervals. For example, assuming that name, height, and weight are columns with large intervals, the additional information after unstructured description processing is as follows: Xiao Ming's height is 180cm, and Xiao Ming's weight is 80KG. After optimizing the long table, optimize the wide table, see Figure 2 ss1.4 in.

[0089] 2) When vectorizing text, formulate different slicing strategies for different types of text and text with different retrieval requirements. The process is as follows:

[0090] 2.1) Preliminarily determine the set slice length based on the maximum number of tokens input to the base model. The calculation formula is as follows:

[0091] L=encding(L h +L prompt +∑L content +∑L seq )

[0092] Among them, L h It is the basic information of the large model, namely, the identity definition of the question-answering assistant, how to respond to uncertain answers, its own identity information (such as whether the model is developed by a certain company), and the length of sentences used to improve the human-computer interaction experience; L prompt is the length of the sentence after the special prompt formulated for this question and the filled-in question is combined; ∑L conent It is the total length of the text in the RAG as a large model prompt paragraph. For example, if the maximum length of each slice is 500, and the system uses the 5 slices most relevant to the question as reference knowledge, it will take up 5*500=2500 text lengths. ∑L seqIt is the total length of the context sentences of the multi-round dialogue supported by the large model. For example, if the number of rounds of multi-round dialogue supported is 3, and the estimated length of each answer + question sentence is 300, then 300*3=900 words are required. After adding, it is encoded through the enbedding model to convert the length of the Chinese sentence into the number of tokens, including the text content of the sentence will be encoded, and some special identifiers will be added as the start and end marks of the sentence. The sum of the common token numbers of the two parts is the total token number overhead required, which needs to be less than the maximum token number of the base large model. In the above variables, ∑L seq The size of is the most difficult to estimate. If the maximum length of the slice is set improperly, combined with ∑L seq Sudden changes may cause video memory overflow, so when setting the maximum length of the slice, you need to leave enough margin to prevent the total number of tokens from exceeding the tolerance limit of LLM.

[0093] 2.2) Based on the above rules, select the maximum length of a single slice, and take 500 as an example. When slicing a document, you need to pay attention not only to the maximum length, but also to set a reasonable overlap length. That is, in order to make the semantics of a single slice more complete and better preserve the contextual relationship, adjacent slices will have a small amount of text overlap. The text overlap length should not be too large, and 20 is taken as an example.

[0094] Corresponding fragmentation strategies need to be formulated for different types of special texts, which will help improve the accuracy of subsequent knowledge retrieval.

[0095] 2.3) Content-based slicing strategy

[0096] When facing markdown type documents, the content of the same level of title should be kept in the same slice as much as possible when slicing, so as to minimize the loss of information caused by the partition of the original content due to document slicing. Just as the content of the first-level title cannot be completely put into the same slice, the same sentence and paragraph should be kept in the same slice as much as possible when slicing. The mark to distinguish whether it is the same sentence or paragraph is a special symbol, such as comma, line break, etc. When setting the text overlap length, since this parameter is generally small and can hardly cover a paragraph or sentence, the mainstream solutions on the market are to set the length of the overlapping text content directly according to the maximum length: 20. This method adopts a dynamic setting method: if the overlap length is too short to cover the sentence of the previous slice, the title corresponding to the slice in the original text will be used as context information and inserted into the content of the slice as a summary of the context information, and the remaining overlap margin will be set according to the mainstream method.

[0097] 3) Construction and indexing of vector library

[0098] This method proposes a construction plan for the data design and search index setting of the vector library based on actual industrial scenarios, taking into account the construction cost, vector library performance, architecture scalability, and ease of use.

[0099] 3.1) Vector library selection

[0100] This method is based on the data characteristics of the industrial industry. There are many sub-industries, high professional barriers between industries, and low knowledge sharing. Even for the same sub-industry, different factories will have different "specific rules". The data of each industry includes two parts: general and private. The cross-overlap between data is relatively high, and the search scope involved is complex. It is precisely because of the detailed data division that the total amount of data for each factory is not particularly high, and not much knowledge can enter the general knowledge base.

[0101] This experiment uses qdrant as the database, whose largest storage unit is collection, which corresponds to the database in the relational database. This method is different from other mainstream methods in that the standard for dividing collections is the embedding model or split_size mentioned above, and does not include the data isolation of different industries in business logic. In this method, the latter is distinguished by the payload field. The advantages of this method over the traditional business logic-based database and table division are that, first, the data division standard of the upper layer of business logic-the different specifications of text vectorization are well integrated into the overall system; second, based on the characteristics of different queries, under the same vectorization standard, the query scope based on business logic division is diverse. If strong data isolation at the collection level is added, a large number of cross-collection queries will appear under the diverse search scope, and the query efficiency is poor. Data between different standards will not be put together for query at all, and they are independent of each other when querying, so the strongest data isolation type collection is selected.

[0102] 3.2)Payload field selection

[0103] According to 3.1, the payload in this method is an important means to distinguish industries and topics, and needs to contain rich resources other than content.

[0104] The payload options of this method include industry, paragraph id, content source, content source ID, pdf-ocr anchor box, whether to enable, article external links, etc. Unique IDs are assigned to paragraphs and article sources. This ID is generated based on the document name and paragraph content. The document IDs are consistent in different collections, which is similar to the search of the same document with different vectorization standards mentioned above. At the same time, the generated unique ID can also be used as a field for building a funny index to optimize the search.

[0105] 3.3) Index settings

[0106] This method has a large number of payload fields, and different fields are required for screening at different search stages. In order to improve retrieval efficiency, indexes are added to the following fields: industry, content source ID, whether enabled, and knowledge base ID fields. The reasons for this setting are as follows:

[0107] For industry fields, whether to enable fields, and knowledge base ID fields, they need to be screened when performing vector searches. Industry fields and knowledge base ID fields involve many values, and the demand for indexes is relatively large. When performing vectorized searches, user questions need to select a specific industry, and then confirm the scope of the knowledge base to be used based on user permissions and model binding knowledge bases. In summary, indexes need to be added to these fields. Whether to enable fields has fewer values. In experiments, this method proves that establishing a joint index of multiple fields is more effective.

[0108] 3.5) Customized hot and cold separation based on vector slicing strategy

[0109] When searching for vectors, as time goes by, the accumulated data will increase, and the corresponding search time will also increase. The data needs to be segmented. This method uses the hot and cold separation technology to put the frequently searched paragraphs into a new collection, hereinafter referred to as the hot library; the old collection is hereinafter referred to as the cold library. If the paragraphs searched in the hot library cannot meet the needs of intelligent question answering, the similarity search is performed in the cold library. The sorting process is as follows: Figure 3 As shown, the detailed steps are as follows:

[0110] 3.5.1) Heat library establishment and data selection

[0111] During the user search, k paragraphs with high similarity will be screened out. The screened paragraphs are divided into two types. One is the core content paragraphs. In this case, paragraphs of different lengths will be involved. These paragraphs have the same content source ID and are processed as the same batch of data during the hot and cold exchange process. That is, if the search frequency is too low, they will all be deleted from the hot database. Both different result types are stored in the hot database and marked by the payload and flag fields to distinguish whether they are core content or non-core content. As the question and answer process proceeds, the search results will accumulate. In order to maintain the system's efficient search and screening of the hot database, according to the LRU mechanism, the relatively less commonly used paragraphs in the hot database are selected according to the principle of the smallest recent paragraph. When removing paragraphs, all those with the same content source ID are removed.

[0112] 3.5.2) Thermal storage data failure criteria

[0113] Due to the particularity of vector library search, each time the hot library is matched for similarity, search results will be obtained, and due to the existence of the re-ranking algorithm, the absolute value of the similarity cannot be obtained, so a threshold cannot be set for judgment. This method uses the ability of the large model. After the search is completed, a specific prompt word is formulated to judge the correlation between the retrieved paragraph and the user input question, and a score between 1 and 100 is obtained. The threshold is set according to this score. If the score obtained by the large model is lower than the threshold, the search knowledge base is switched to the cold library, and the above operation is repeated. If the search results of the cold library are significantly improved compared to the hot library, the search knowledge of the cold library is selected.

[0114] 3.5.3) Pre-search

[0115] Due to the data isolation between the hot and cold repositories and the low cache hit rate, the search time will be longer. Based on the particularity of vector search and the capabilities of large models, this method proposes the concept of pre-retrieval. Every time a popular paragraph is stored in the hot database, the user question is cached at the same time. The user question and the related paragraph have the same life cycle, that is, when the related knowledge is removed from the hot database, the cached user question is also removed. Before searching the hot database, the user question and the cached related question are calculated for similarity through the large model, and a threshold is set. Only when the value is higher than the threshold will the operation of 3.5.2 be performed to start searching for knowledge from the hot database. Since the user question is short and the similarity comparison speed is fast, the retrieval speed can be greatly reduced when the cache hit rate is not high.

[0116] 3.6) Vector library search

[0117] Based on the description of the above two points, at the beginning of the search, a similarity search of the initial vectorization parameter collection will be performed first. After the searched structures are rearranged, relevant paragraphs are searched in collections of different vectorization standards according to different document IDs, and then integrated, and finally the answer content is output.

[0118] When using the SDK to connect to the vector library, you need to instantiate the client, and the same client can only connect to the same collection.

[0119] Therefore, all the data of this patent is stored in one collection, and metadata is used to distinguish different industry fields. Metadata is the object storage function of qdrant, which can store structured or unstructured data. A field in metadata is used to distinguish different domain knowledge, and an index is added to the field to ensure query efficiency.

[0120] Embodiment 3 is an embodiment of the present invention, which provides a knowledge base construction system for a large model, including:

[0121] Text preprocessing module, text slicing module and knowledge base building module;

[0122] The text preprocessing module is used to preprocess multi-source heterogeneous files, extract and standardize the content;

[0123] The text slicing module is used to slice and vectorize text to retain context information;

[0124] The knowledge base building module is used to store vectorized data into the vector library and perform indexing and optimization.

[0125] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0126] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0127] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0128] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0129] Example 4 is an embodiment of the present invention, which provides a method for constructing a knowledge base of a large model. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.

[0130] In order to verify the effect, the present invention conducts experiments in a simulated environment, and writes corresponding test scripts to verify and determine the efficacy of the invention in the entire system. The specific conditions of the environment are as follows:

[0131] The number of documents in the knowledge base is 120,000, the average length of the slice is 742, and there are a total of 1.1 million nodes. Before the launch of the present invention, user questions were randomly collected, and the average time for a single query was 53ms for 100 queries. After building the hot and cold libraries, the average query time for a single query was 32ms for 100 queries, and the cache hit rate was 66%, which is significantly improved compared to the traditional architecture.

[0132] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for constructing a knowledge base of a large model, characterized in that: include: Preprocess multi-source heterogeneous files, extract and standardize content; Slice and vectorize text to preserve contextual information; Store vectorized data into the vector library and perform indexing and optimization.

2. The method for constructing a knowledge base of a large model as claimed in claim 1, characterized in that: The multi-source heterogeneous files include structured data files, semi-structured data files and unstructured data files; The extracting and standardizing the content includes directly extracting text content from the structured data file; Extract text content from semi-structured data files and convert format information into a unified markup format; Use optical character recognition technology to extract text from unstructured data files and parse and process special format content in documents.

3. The method for constructing a knowledge base of a large model as claimed in claim 2, characterized in that: The special format content in the document includes detecting table areas, and using an object detection model to identify tables with and without borders; For bordered tables, identify cell borders and perform OCR processing; For borderless tables, use OCR recognition directly to determine the row and column relationship of the text through the anchor box coordinates; For table data that spans pages or slices, extract the header information and keep the header in the slice; Optimize wide tables and strengthen relationships between distant columns through unstructured descriptions.

4. The method for constructing a knowledge base of a large model as claimed in claim 3, characterized in that: The text slicing and vectorizing includes preliminarily determining the set slice length according to the maximum number of input tokens of the base large model, and formulating corresponding slicing strategies for different types of special texts.

5. The method for constructing a knowledge base of a large model as claimed in claim 4, characterized in that: The slice length is expressed as, L=encding(L h +L prompt +∑L content +L seq ) Among them, L h is the basic information of the large model, L prompt is the length of the sentence after the special prompt formulated for this question and the filled-in question is combined, ∑L connant is the total length of the text in the RAG as a large model prompt paragraph, ∑L seq is the total length of context sentences in multi-round conversations supported by the large model, and encding represents the encding model.

6. The method for constructing a knowledge base of a large model as claimed in claim 5, characterized in that: The slicing strategy includes, for Markdown documents, giving priority to putting the contents under the same level heading into the same slice when slicing; If it cannot be accommodated completely, keep the sentence or paragraph in the same slice; The length of the overlapping part of the slices is dynamically adjusted according to the semantic structure of the text. If the overlapping part is not enough to cover the sentence of the previous slice, the original title information is inserted into the slice content.

7. The method for constructing a knowledge base of a large model as claimed in claim 6, characterized in that: The storing of vectorized data into a vector library, performing indexing and optimization includes selecting a vector database based on data characteristics in the industrial field, wherein the minimum storage unit is a collection, dividing different collections according to a text vectorization model or a slice size, and distinguishing data through a payload field; Select the payload field to distinguish industries and topics. The fields include industry, paragraph ID, content source ID, and create indexes for the fields. Use hot and cold separation technology to store frequently accessed data segments in hot storage, and infrequently accessed data in cold storage. When searching, search in hot storage first. If the demand cannot be met, turn to cold storage. During the search process, a similarity search of the initial vectorized parameters is first performed. After the search results are rearranged, relevant paragraphs are retrieved from different collections based on the document ID. After the search is completed, the final answer output is generated by integrating the various parts of the content. All data is stored in the same collection, and different industries and domain knowledge are distinguished through metadata fields.

8. A system using the knowledge base construction method of the large model as described in any one of claims 1 to 7, characterized in that: include: Text preprocessing module, text slicing module and knowledge base building module; The text preprocessing module is used to preprocess multi-source heterogeneous files and extract and standardize content; The text slicing module is used to slice and vectorize the text to retain context information; The knowledge base construction module is used to store vectorized data into a vector library and perform indexing and optimization.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for constructing a knowledge base of a large model according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for constructing a knowledge base of a large model according to any one of claims 1 to 7 are implemented.

Images