Knowledge base creation and query method

Abstract

The invention relates to the field of knowledge base construction, in particular to a knowledge base creation and query method. The invention aims to improve the efficiency of processing the unstructured long text by the knowledge base system. According to the method, a long text can be segmented into small fragments according to semantics or text length and semantic vectors can be generated by adopting text fragmentation of a large language model and semantic vectorization or a semantic vectorization technology of a special text vectorization model and based on fixed-length fragmentation of empirical values, so that context information related to query can be accurately captured. Vector search and a word segmentation function in Elasticsearch are combined, and synonyms, polysemes and complex context association problems are automatically processed. Besides, through the support of a document database and Elasticsearch distributed processing, the system can quickly position related contents under a large data volume, and the problem of low query efficiency of a traditional method on a large-scale data set is solved.

Description

Technical Field

[0001] The present invention relates to the field of knowledge base construction, and in particular to a knowledge base creation and query method. Background Art

[0002] With the development of artificial intelligence and big data technology, knowledge management systems and search technologies have been widely used. Traditional knowledge base construction methods mostly rely on keyword matching and rule retrieval, and typical applications include full-text search engines and question-answering systems. These systems usually perform simple matching between user queries and document content based on document indexes and keyword Boolean query structures.

[0003] Elasticsearch is an open source distributed search and analysis engine, which is usually used to quickly search and analyze large-scale text and structured data. It is built on Apache Lucene, can process real-time data, and is suitable for full-text retrieval, log analysis, real-time monitoring and other operations on massive information.

[0004] In the knowledge base construction scenario, Elasticsearch can be used for the following: a. Full-text search: It can be used to perform full-text search in sharded documents based on user queries to find semantically appropriate shards. Elasticsearch provides efficient full-text search and approximate matching capabilities, and can handle complex queries in text data.

[0005] b. Storage and indexing: Elasticsearch can store a large amount of document data and provide flexible index configuration, which makes it easy to set weights for different fields. Combined with Elasticsearch's index management, documents in the knowledge base can be accurately organized and query results can be optimized as needed.

[0006] Large Language Model (LLM) is a natural language processing (NLP) model based on deep learning. It is usually trained with large-scale text data and can understand and generate natural language. Large language models learn the semantics, syntax, contextual associations, etc. in language through a large number of parameters. They are good at processing tasks such as semantic analysis, text generation, and dialogue understanding. Therefore, they perform well in language understanding and generation and are often used to realize applications such as human-computer dialogue, text summarization, and intelligent question and answer.

[0007] In the construction and query of the knowledge base, the role of the large language model is mainly reflected in the following two aspects: a. Text segmentation and semantic vectorization in knowledge base construction: When building a knowledge base, the large language model can segment unstructured document content (such as long texts, PDFs, etc.) according to preset rules, such as dividing it into small segments by chapter, sentence length, context, etc.

[0008] For each shard, the large language model generates corresponding semantic vectors, which can capture the semantic information and contextual associations in the shard, giving it a high level of semantic expression. This vectorization process allows the shard content to be better recognized and matched by computers, providing support for subsequent efficient semantic queries.

[0009] b. Semantic matching and natural language generation in knowledge base queries: During the user query phase, the large language model converts the query into a semantic vector and compares it with the document segment vectors stored in the knowledge base to find the most semantically relevant segment. This can improve search accuracy and surpass the limitations of traditional keyword matching.

[0010] In addition, after obtaining the most relevant segments, the large language model can organize the key information in the segments into answers close to natural language through the natural language generation function, providing users with more natural and coherent answer results.

[0011] Therefore, the large language model not only improves the semantic understanding and matching capabilities of queries in the knowledge base system, but also significantly improves the quality and expression of query results, enabling the knowledge base system to better meet users' needs for information accuracy and readability.

[0012] Existing knowledge base systems have the following problems when processing unstructured long texts: 1) Low keyword matching accuracy: Keyword-based retrieval methods have difficulty understanding user intent and cannot accurately extract contextual information related to the query, especially for complex problems in natural language processing.

[0013] 2) Low quality of search results: Keyword matching methods cannot cope with synonyms, polysemous words, and language context differences, resulting in inaccurate search results.

[0014] 3) Unable to process massive amounts of information: Traditional retrieval methods are inefficient when querying large-scale data sets and cannot quickly obtain the most relevant content.

[0015] These problems mainly stem from the fact that traditional search methods cannot fully utilize text semantic information and contextual associations, which affects the efficiency and quality of users' knowledge acquisition. Summary of the invention

[0016] The present invention aims to improve the efficiency and effectiveness of the knowledge base system in processing unstructured long texts, especially to solve the shortcomings of the existing technology in keyword matching, search result quality and massive data processing. Specifically, the defects of the current system are improved item by item through the following innovative means: 1. Improve the accuracy of keyword matching: Using text segmentation and semantic vectorization of a large language model or semantic vectorization technology of a dedicated text vectorization model, fixed-length segmentation based on experience values, it is possible to segment long text into small segments according to semantics or text length, and generate semantic vectors, thereby accurately capturing contextual information related to the query. In this way, the accuracy of text matching can be greatly improved, overcoming the problem of difficulty in understanding user intent in existing technologies.

[0017] 2. Improve the accuracy of search results: Combining vector search with the word segmentation function in Elastic Search can automatically handle synonyms, polysemous words, and complex context associations. In this way, the system can more accurately find the most relevant document content, effectively solving the defect of low search result quality in traditional keyword matching methods.

[0018] 3. Efficient processing of massive data: Introducing vectorized processing in the knowledge base construction and query phase can improve retrieval speed by using vector search technology. In addition, with the support of document database and Elastic Search distributed processing, the system can quickly locate relevant content under large amounts of data, solving the problem of low query efficiency of traditional methods on large-scale data sets.

[0019] Through the above means, the present invention aims to achieve accurate and efficient semantic retrieval and information acquisition of massive document contents in the knowledge base, so that users can quickly obtain the answers they need and improve the convenience and accuracy of knowledge acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a method flow diagram of an implementation method of creating a knowledge base using a large language model, a relational database, a document database and a vector database as described in the present invention.

[0021] Figure 2 It is a schematic diagram of the specific algorithm flow of the present invention. DETAILED DESCRIPTION

[0022] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0023] The implementation method of creating a knowledge base using a large language model, relational database, document database and vector database.

[0024] The present invention realizes efficient and accurate semantic retrieval in the construction and query of knowledge base through the combined application of multiple innovative technologies, thereby solving the shortcomings of the existing technology in terms of keyword matching accuracy, search result quality and massive data processing. Figure 1 ): (I) Text segmentation technology: Text segmentation is achieved through a large language model, and long texts are segmented according to preset segmentation rules, including by chapter, fixed length, fixed delimiter, or by using the natural segmentation ability of the large language model to generate small segments. After semantic segmentation, each segment can capture contextual information and enhance the semantic understanding ability during query. This technology is used to segment documents and store them in the knowledge base, effectively solving the problem of low accuracy of keyword matching.

[0025] (II) Semantic vector calculation technology: Based on text segmentation, vector calculation technology is used to generate a corresponding semantic vector for each segment, and the vector is associated with the segment number and the document to which it belongs and stored in the database. This semantic vector can encode the deep semantic information of the text, allowing the system to perform query matching based on semantic similarity. This method solves the quality problem of search results and can process the differences in synonyms, polysemous words and language context through semantic association, making the query results more accurate.

[0026] (III) Retrieval technology combining vector search and word segmentation: In the user query stage, the system first converts the user's query into a semantic vector, and then uses vector search technology to find the document fragment with the closest semantics from the knowledge base. Then, the query results are further screened using the word segmentation function of ElasticSearch, and the found fragments and their original documents are stored in a temporary database. This combination of technologies significantly improves the processing efficiency of large data sets and solves the problem of low query efficiency of traditional methods in massive data.

[0027] (IV) Natural language generation technology: Based on the search results, the queried fragments and original document content are input into a large language model, and the model generates answers that conform to natural language expression to improve user experience. This natural language generation technology effectively enhances the readability of search results, allowing users to obtain clearer and more coherent answers.

[0028] Through the organic combination of the above-mentioned technologies, the present invention provides a complete knowledge base construction and query method, which can achieve higher keyword matching accuracy, improve the quality of search results, and significantly optimize the efficiency of large-scale data query when processing unstructured long texts, thereby meeting users' needs for efficient and accurate information acquisition from the knowledge base system.

[0029] Compared with the prior art, the present invention has the following advantages: 1. Improve retrieval accuracy: By introducing a large language model and vector computing technology, we can better understand the semantic intent of user queries and provide retrieval results that are closer to the context; at the same time, we use ElasticSearch to store and quickly retrieve relevant content, further improving retrieval accuracy.

[0030] 2. Enhanced search efficiency: Using vector search technology, the most relevant document fragments can be quickly located in massive data, greatly shortening the query time.

[0031] 3. Improve the quality of results: With the power of large language models, search results can be organized and expressed in a more natural language form, allowing users to obtain answers that are easier to understand.

[0032] 4. Strong scalability: This method can process various types of documents, and through parallel processing technology, it is suitable for the construction and query of large-scale document libraries, and has broad application prospects.

[0033] Example 1: Import several text files into the system, set the sharding rule to shard every 700 characters, store the shards and documents in the document database and ElasticSearch, use the pre-trained model based on the ChatGLM3-6B model to vectorize each shard, and store the results in the vector database. When the user queries, after entering the question, the relevant shards are found through semantic vector matching, and the final answer is generated by combining the large language model and the ElasticSearch query results.

[0034] Embodiment 2: An electronic document containing multiple chapters is segmented at the chapter level, the segments and documents are stored in a document database and ElasticSearch, and a specific topic model is used to perform vector calculations to generate topic vectors for the documents. When a user queries, matching is performed based on the topic and explanatory answers are generated, and answers are generated in combination with the ElasticSearch query results.

[0035] In the storage medium, the computer program may include the process of the embodiments of the above methods when executed. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0036] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for creating a knowledge base, characterized in that: The method comprises: (1.1) Segment the knowledge document to obtain several fragments; (1.2) Use vector computing technology to generate the semantic vector corresponding to each shard and obtain the shard semantics; (1.3) Store shard semantics, shards, and knowledge documents in an associated manner to complete the knowledge base construction.

2. The knowledge base creation method according to claim 1, characterized in that: In step (1.1), the knowledge document is segmented by chapters, fixed lengths, fixed delimiters, or by utilizing the segmentation capabilities of a large language model.

3. A method for querying a knowledge base created according to the method of claim 1, characterized in that: include: (3.1) Convert the query input by the user into a semantic vector to obtain the query semantics; (3.2) Use vector search technology to find several shard semantics closest to the query semantics from the knowledge base; (3.3) Combine the word segmentation function of Elasticsearch to further screen the shards with the closest semantic correspondence in (3.2); (3.4) Input the fragments obtained by (3.3) and their corresponding knowledge documents into the large language model to generate answers that conform to natural language expression.

4. A knowledge base creation system, characterized in that: include: The sharding module is used to segment the knowledge document into several shards; A vector calculation module is used to generate a semantic vector corresponding to each shard using vector calculation technology to obtain shard semantics; The data storage module is used to associate and store shard semantics, shards, and knowledge documents.

5. Retrieval module, used to realize semantic retrieval by combining vector search and word segmentation technology.

6. A knowledge base query system, characterized in that: include: An input module is used to accept user queries; The vector calculation module is used to convert the query input by the user into a semantic vector to obtain the query semantics; A retrieval module is used to find several shard semantics closest to the query semantics from the knowledge base using vector search technology; The screening module is used to further screen the shards with the closest semantic correspondence in (3.2) by combining the word segmentation function of Elasticsearch; The output module is used to input the screened fragments and their corresponding knowledge documents into the large language model to generate answers that conform to natural language expression.

7. A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, characterized in that: When the instructions are executed by a computing device, the computing device is caused to perform the method as claimed in claim 3.

8. An electronic device, characterized in that: The method comprises one or more processors, one or more memories and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing the method as claimed in claim 3.

Images