Knowledge base construction method and system based on large language model

Abstract

The present invention provides a method and system for constructing a knowledge base based on a large language model, the method comprising: preprocessing the original text to obtain a plurality of original text paragraphs; deeply elaborating each of the original text paragraphs through a language enhancement model to generate detailed text content, and converting the detailed text content into a high-dimensional vector through a text vector model, wherein the high-dimensional vector matches one of the original text paragraphs; and storing the high-dimensional vector and the corresponding original text paragraph in a database, wherein the database supports vector indexing and retrieval operations. Through this application, the productivity of text processing can be improved, which is conducive to saving a certain amount of manpower and material resources, and is also conducive to reducing the difficulty of database maintenance.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge base construction, and in particular to a knowledge base construction method and system based on a large language model. Background Art

[0002] At present, the construction of knowledge base mainly relies on manual methods to segment and summarize text documents, store the text content in the database, and directly present the searched content on the front end through keyword retrieval, or add fixed content introductions before and after to enhance text comprehension.

[0003] Traditional knowledge bases are primarily built to process text data, with limited capabilities for integrating and processing multimodal data such as images and audio. The traditional knowledge base construction process requires manual knowledge extraction, summarization, and storage, a time-consuming, labor-intensive, and error-prone process. At the same time, with the continuous updating and changes in knowledge, knowledge base maintenance has also become a huge challenge. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to provide a knowledge base construction method and system based on a large language model to address the deficiencies in the prior art.

[0005] To achieve the above objectives, the present invention provides a method for constructing a knowledge base based on a large language model, the method comprising:

[0006] Preprocessing the original text to obtain multiple original text paragraphs;

[0007] Performing an in-depth explanation of each of the original text paragraphs using a language enhancement model to generate detailed text content, and converting the detailed text content into a high-dimensional vector using a text vector model, wherein the high-dimensional vector matches one of the original text paragraphs;

[0008] Storing the high-dimensional vector and the corresponding original text paragraph in a database, wherein the database supports vector indexing and retrieval operations;

[0009] Converting the query question into a question vector, comparing the question vector with the text vectors in the database, and filtering N relevant text paragraphs related to the question vector from the database based on the comparison result;

[0010] The step of selecting N relevant text paragraphs related to the question vector from the database according to the comparison result includes:

[0011] Calculating the question vector and each high-dimensional vector in the database one by one using a cosine similarity calculation method to obtain a similarity score between the question vector and each high-dimensional vector;

[0012] N relevant text paragraphs related to the question vector are screened out based on the multiple similarity scores.

[0013] Preferably, after screening out N relevant text paragraphs related to the question vector based on the plurality of similarity scores, the method further comprises:

[0014] The N relevant text paragraphs are sorted by a sorting model, a weight value is assigned to each relevant text paragraph according to the sorting result, and the final relevant text paragraph is screened out based on each weight value, wherein the weight value is used to reflect the degree of relevance between the relevant text paragraph and the corresponding question to be queried.

[0015] Preferably, after screening out the final relevant text paragraphs based on the weight values, the method further includes:

[0016] Inputting the final relevant text paragraph and the corresponding weight value into the language enhancement model to generate a target answer;

[0017] The target answer is displayed through a user interface.

[0018] Preferably, the step of preprocessing the original text to obtain a plurality of original text paragraphs includes:

[0019] The original text is obtained and segmented using natural language processing technology to obtain multiple independent original text paragraphs.

[0020] To achieve the above objectives, the present invention further provides a knowledge base construction system based on a large language model, which is used to implement the above-mentioned knowledge base construction method based on a large language model. The system includes:

[0021] A preprocessing module, used for preprocessing the original text to obtain multiple original text paragraphs;

[0022] a conversion module, configured to perform an in-depth elaboration on each of the original text paragraphs using a language enhancement model to generate detailed text content, and convert the detailed text content into a high-dimensional vector using a text vector model, wherein the high-dimensional vector matches one of the original text paragraphs;

[0023] A storage module, configured to store the high-dimensional vector and the corresponding original text paragraph in a database, wherein the database supports vector indexing and retrieval operations;

[0024] a comparison module, configured to convert a query question into a question vector, compare the question vector with text vectors in the database, and select N relevant text paragraphs related to the question vector from the database based on the comparison result;

[0025] The comparison module includes:

[0026] a calculation unit, configured to calculate the similarity scores between the question vector and each high-dimensional vector in the database one by one by using a cosine similarity calculation method;

[0027] A screening unit is configured to screen out N relevant text paragraphs related to the question vector based on the multiple similarity scores.

[0028] Preferably, after the screening unit, the system further comprises:

[0029] A sorting unit is used to sort the N relevant text paragraphs through a sorting model, assign a weight value to each relevant text paragraph according to the sorting result, and screen out the final relevant text paragraph based on each weight value, wherein the weight value is used to reflect the degree of relevance between the relevant text paragraph and the corresponding question to be queried.

[0030] Preferably, after the sorting unit, the system further includes:

[0031] A generating unit, configured to input the final relevant text paragraph and the corresponding weight value into the language enhancement model to generate a target answer;

[0032] A display unit is used to display the target answer through a user interface.

[0033] Preferably, the preprocessing module includes:

[0034] The segmentation unit is used to obtain the original text and segment the original text using natural language processing technology to obtain multiple independent original text paragraphs.

[0035] The beneficial effects of the present invention are: by preprocessing the original text, multiple original text paragraphs are obtained, each original text paragraph is deeply elaborated through the language enhancement model to generate detailed text content, and the detailed text content is converted into a high-dimensional vector through the text vector model, and then the high-dimensional vector and the corresponding original text paragraph are stored in the database. Different from the existing technology, it can improve the productivity of text processing, which is conducive to saving a certain amount of manpower and material resources, and at the same time it is also conducive to reducing the difficulty of database maintenance. The application of the language enhancement model can further enhance the context of the text, making the information in the database more accurate and complete, which is conducive to improving the overall quality of the database and providing a more solid foundation for subsequent knowledge retrieval.

[0036] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of a method for constructing a knowledge base based on a large language model provided in the first embodiment of the present invention;

[0038] Figure 2 This is a structural block diagram of a knowledge base construction system based on a large language model provided in the second embodiment of the present invention.

[0039] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts are within the scope of protection of this application.

[0041] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can, without inventive effort, apply the present application to other similar scenarios based on these drawings. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, changes in design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0042] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0043] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "one", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The word "multiple" used in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0044] See also Figure 1 , is a flowchart of a method for constructing a knowledge base based on a large language model in a first embodiment of the present invention, the method for constructing a knowledge base based on a large language model comprising the following steps:

[0045] Step S101, preprocessing the original text to obtain multiple original text paragraphs;

[0046] The step of preprocessing the original text to obtain multiple original text paragraphs includes:

[0047] The original text is obtained and segmented using natural language processing technology to obtain multiple independent original text paragraphs.

[0048] It should be noted that the input original text (such as books, articles, reports and other documents in various formats) is cut into a series of independent text paragraphs through natural language processing technology. This step helps to break down large blocks of text content into smaller units that are easier to process.

[0049] Step S102: each of the original text paragraphs is deeply elaborated using a language enhancement model to generate detailed text content, and the detailed text content is converted into a high-dimensional vector using a text vector model, wherein the high-dimensional vector matches one of the original text paragraphs;

[0050] Among them, the language enhancement model can understand and generate text content that is similar to the original text, contains more details or is easier to understand. In this step, the original text paragraph is deeply explained through the language enhancement model (such as the GPT series), which can retain and enhance the context of the original text paragraph. This not only helps to improve the accuracy of the database, but also enhances the readability and usability of the data in the database.

[0051] It should be noted that advanced text vector models (such as embeddings) are used to convert text paragraphs processed by the language enhancement model into high-dimensional vectors. These high-dimensional vectors can capture the semantic features and contextual relationships of the detailed text content, providing a basis for subsequent knowledge retrieval.

[0052] Step S103: storing the high-dimensional vector and the corresponding original text paragraph in a database, wherein the database supports vector indexing and retrieval operations.

[0053] The database should be selected to support efficient vector indexing and retrieval operations so as to quickly find knowledge related to the query.

[0054] Through the above steps, the original text is preprocessed to obtain multiple original text paragraphs, and each original text paragraph is deeply elaborated through the language enhancement model to generate detailed text content, and the detailed text content is converted into a high-dimensional vector through the text vector model. Then, the high-dimensional vector and the corresponding original text paragraph are stored in the database. Different from the existing technology, it can improve the productivity of text processing, help save a certain amount of manpower and material resources, and also help reduce the difficulty of database maintenance. The application of the language enhancement model can further enhance the context of the text, making the information in the database more accurate and complete, which is conducive to improving the overall quality of the database and providing a more solid foundation for subsequent knowledge retrieval.

[0055] In some embodiments, the method further comprises:

[0056] Pre-train based on the large language model to obtain the initial generation model;

[0057] Among them, first, the traditional large language model is used as the technical foundation for basic pre-training, so that the initial generation model can generate reasonable text. The specific steps of pre-training based on the large language model are to train the traditional large language model through large-scale historical text data to learn the basic structure and vocabulary of the language.

[0058] Customizing a reward function and performing reinforcement training on the initial generation model using the reward function to obtain a preliminary language reinforcement model;

[0059] In order to make the text generated by the initial generative model more in line with the target, a reward function is designed. This reward function takes into account multiple dimensions, such as the accuracy, fluency, information richness and relevance of the generated answers to the expected goals. For example, in question-answering tasks, the model can receive corresponding rewards based on factors such as the accuracy, completeness and interpretability of the answers.

[0060] Specifically, during training, the model receives rewards or penalties based on the generated text, and then adjusts its generation strategy. This process is achieved through methods such as "policy gradient" or "Q-learning" to maximize the final long-term reward.

[0061] The preliminary language reinforcement model is optimized through a feedback mechanism to obtain the required language reinforcement model.

[0062] After each generated text output, a reinforcement learning mechanism optimizes the model's behavior through feedback (evaluated from a reward function). Through trial and error, and by receiving rewards and penalties, the model continuously adjusts its generation strategy to improve the quality and consistency of generated text. Ultimately, after multiple rounds of reinforcement learning training and optimization, the model generates even higher-quality answers, ensuring fluency while also enhancing the accuracy, richness, and reasoning power of responses.

[0063] In some embodiments, after storing the high-dimensional vector and the corresponding original text paragraph in a database, the method further includes:

[0064] The query question is converted into a question vector, the question vector is compared with the text vectors in the database, and N relevant text paragraphs related to the question vector are screened out from the database according to the comparison result.

[0065] Among them, when the user inputs a question to be queried, the text vector model in step S102 is first used to convert the question to be queried into a question vector. The question vector and the high-dimensional vector are both text vectors. This step ensures that the question to be queried can be compared with the high-dimensional vector in the database in the same semantic space so that the similarity score between them can be calculated. In addition, it also realizes accurate vectorization processing of the query question and efficient comparison with the text vector in the database. Compared with the traditional knowledge retrieval method of searching by keyword matching, it is not only beneficial to improve the accuracy of knowledge retrieval, but also can greatly shorten the retrieval time and improve the retrieval efficiency, thereby improving the user experience.

[0066] It should be noted that, based on the comparison results, N original text paragraphs related to the question to be queried are screened out. The value of N can be adjusted according to actual needs to balance the retrieval accuracy and recall rate, and the original text paragraphs related to the question to be queried are the relevant text paragraphs.

[0067] In some embodiments, the step of selecting N relevant text paragraphs related to the question vector from the database according to the comparison results includes:

[0068] Calculating the question vector and each high-dimensional vector in the database one by one using a cosine similarity calculation method to obtain a similarity score between the question vector and each high-dimensional vector;

[0069] N relevant text paragraphs related to the question vector are screened out based on the multiple similarity scores.

[0070] In some embodiments, after selecting N relevant text paragraphs related to the question vector based on the plurality of similarity scores, the method further includes:

[0071] The N relevant text paragraphs are sorted by a sorting model, a weight value is assigned to each relevant text paragraph according to the sorting result, and the final relevant text paragraph is screened out based on each weight value, wherein the weight value is used to reflect the degree of relevance between the relevant text paragraph and the corresponding question to be queried.

[0072] Among them, the sorting model is a sorting algorithm based on machine learning, and the sorting model can be trained and optimized according to the semantic features, contextual relationships and user feedback information of the text to update the sorting model in real time to improve the accuracy of the sorting model.

[0073] It should be noted that the weight value assigned to the relevant text paragraph can reflect the degree of relevance between the relevant text paragraph and the question to be queried.

[0074] In some embodiments, after selecting the final relevant text paragraphs based on the weight values, the method further includes:

[0075] Inputting the final relevant text paragraph and the corresponding weight value into the language enhancement model to generate a target answer;

[0076] Among them, the language enhancement model can comprehensively consider the information of multiple related text paragraphs. Therefore, the final related text paragraphs and corresponding weight values are input into the language enhancement model to perform language reorganization, so as to obtain a coherent and accurate target answer.

[0077] The target answer is displayed through a user interface.

[0078] The second embodiment of the present application also provides a knowledge base construction system based on a large language model, which is used to implement the above-mentioned first embodiment and preferred implementation mode, and will not be repeated here. As used below, the terms "module", "unit", "sub-unit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0079] like Figure 2 As shown in FIG. 1 , it is a structural block diagram of a knowledge base construction system based on a large language model in the second embodiment of the present application. Figure 2 As shown, the system includes:

[0080] A preprocessing module 10 is used to preprocess the original text to obtain multiple original text paragraphs;

[0081] a conversion module 20 for deeply elaborating each of the original text paragraphs using a language enhancement model to generate detailed text content, and converting the detailed text content into a high-dimensional vector using a text vector model, wherein the high-dimensional vector matches one of the original text paragraphs;

[0082] The storage module 30 is configured to store the high-dimensional vector and the corresponding original text paragraph in a database, wherein the database supports vector indexing and retrieval operations.

[0083] In specific implementation, the original text is preprocessed to obtain multiple original text paragraphs, and each original text paragraph is deeply elaborated through the language enhancement model to generate detailed text content, and the detailed text content is converted into a high-dimensional vector through the text vector model. The high-dimensional vector and the corresponding original text paragraph are then stored in the database. Different from the existing technology, this method can improve the productivity of text processing, save a certain amount of manpower and material resources, and reduce the difficulty of database maintenance. The application of the language enhancement model can further enhance the context of the text, making the information in the database more accurate and complete, which is conducive to improving the overall quality of the database and providing a more solid foundation for subsequent knowledge retrieval.

[0084] In some embodiments, after the storage module 30, the system further includes:

[0085] The comparison module is used to convert the query question into a question vector, compare the question vector with the text vectors in the database, and filter out N relevant text paragraphs related to the question vector from the database according to the comparison result.

[0086] In some embodiments, the comparison module includes:

[0087] a calculation unit, configured to calculate the similarity scores between the question vector and each high-dimensional vector in the database one by one by using a cosine similarity calculation method;

[0088] A screening unit is configured to screen out N relevant text paragraphs related to the question vector based on the multiple similarity scores.

[0089] In some embodiments, after the screening unit, the system further comprises:

[0090] A sorting unit is used to sort the N relevant text paragraphs through a sorting model, assign a weight value to each relevant text paragraph according to the sorting result, and screen out the final relevant text paragraph based on each weight value, wherein the weight value is used to reflect the degree of relevance between the relevant text paragraph and the corresponding question to be queried.

[0091] In some embodiments, after the screening unit, the system further comprises:

[0092] A generating unit, configured to input the final relevant text paragraph and the corresponding weight value into the language enhancement model to generate a target answer;

[0093] A display unit is used to display the target answer through a user interface.

[0094] In some embodiments, the pre-processing module 10 includes:

[0095] The segmentation unit is used to obtain the original text and segment the original text using natural language processing technology to obtain multiple independent original text paragraphs.

[0096] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0097] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for constructing a knowledge base based on a large language model, characterized in that: The method comprises: Preprocessing the original text to obtain multiple original text paragraphs; Performing an in-depth explanation of each of the original text paragraphs using a language enhancement model to generate detailed text content, and converting the detailed text content into a high-dimensional vector using a text vector model, wherein the high-dimensional vector matches one of the original text paragraphs; Storing the high-dimensional vector and the corresponding original text paragraph in a database, wherein the database supports vector indexing and retrieval operations; Converting the query question into a question vector, comparing the question vector with the text vectors in the database, and filtering N relevant text paragraphs related to the question vector from the database based on the comparison result; The step of selecting N relevant text paragraphs related to the question vector from the database according to the comparison result includes: Calculating the question vector and each high-dimensional vector in the database one by one using a cosine similarity calculation method to obtain a similarity score between the question vector and each high-dimensional vector; N relevant text paragraphs related to the question vector are screened out based on the multiple similarity scores.

2. The method for constructing a knowledge base based on a large language model according to claim 1, characterized in that: After screening out N relevant text paragraphs related to the question vector based on the plurality of similarity scores, the method further includes: The N relevant text paragraphs are sorted by a sorting model, a weight value is assigned to each relevant text paragraph according to the sorting result, and the final relevant text paragraph is screened out based on each weight value, wherein the weight value is used to reflect the degree of relevance between the relevant text paragraph and the corresponding question to be queried.

3. The method for constructing a knowledge base based on a large language model according to claim 2, characterized in that: After the final relevant text paragraphs are screened out based on the weight values, the method further includes: Inputting the final relevant text paragraph and the corresponding weight value into the language enhancement model to generate a target answer; The target answer is displayed through a user interface.

4. The method for constructing a knowledge base based on a large language model according to claim 1, characterized in that: The step of preprocessing the original text to obtain multiple original text paragraphs includes: The original text is obtained and segmented using natural language processing technology to obtain multiple independent original text paragraphs.

5. A knowledge base construction system based on a large language model, used to implement the knowledge base construction method based on a large language model according to any one of claims 1 to 4, characterized in that: The system comprises: A preprocessing module, used for preprocessing the original text to obtain multiple original text paragraphs; a conversion module, configured to perform an in-depth elaboration on each of the original text paragraphs using a language enhancement model to generate detailed text content, and convert the detailed text content into a high-dimensional vector using a text vector model, wherein the high-dimensional vector matches one of the original text paragraphs; A storage module, configured to store the high-dimensional vector and the corresponding original text paragraph in a database, wherein the database supports vector indexing and retrieval operations; a comparison module, configured to convert a query question into a question vector, compare the question vector with text vectors in the database, and select N relevant text paragraphs related to the question vector from the database based on the comparison result; The comparison module includes: a calculation unit, configured to calculate the similarity scores between the question vector and each high-dimensional vector in the database one by one by using a cosine similarity calculation method; A screening unit is configured to screen out N relevant text paragraphs related to the question vector based on the multiple similarity scores.

6. The knowledge base construction system based on a large language model according to claim 5 is characterized in that: After the screening unit, the system further comprises: A sorting unit is used to sort the N relevant text paragraphs through a sorting model, assign a weight value to each relevant text paragraph according to the sorting result, and screen out the final relevant text paragraph based on each weight value, wherein the weight value is used to reflect the degree of relevance between the relevant text paragraph and the corresponding question to be queried.

7. The knowledge base construction system based on a large language model according to claim 6 is characterized in that: After the sorting unit, the system further includes: A generating unit, configured to input the final relevant text paragraph and the corresponding weight value into the language enhancement model to generate a target answer; A display unit is used to display the target answer through a user interface.

8. The knowledge base construction system based on a large language model according to claim 5, characterized in that: Before the comparison module, the pre-processing module includes: The segmentation unit is used to obtain the original text and segment the original text using natural language processing technology to obtain multiple independent original text paragraphs.

Images